{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# SVM——糖尿病患者分类"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "数据集只有一个文件（diabetes.csv）：Pima Indians Diabetes Dataset 包括根据医疗 记录的比马印第安人 5 年内糖尿病的发病情况，这是一个两类分类问题。每个类的 样本数目数量不均等。一共有 768 个样本，每个样本有 8 个输入变量和 1 个输出 变量。缺失值通常用零值编码。\n",
    "\n",
    "SVC、SVC + GridSearchCV进行参数调优。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 首先 import 必要的模块\n",
    "import pandas as pd \n",
    "import numpy as np\n",
    "\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "#竞赛的评价指标为logloss\n",
    "#from sklearn.metrics import log_loss  \n",
    "#SVM并不能直接输出各类的概率，所以在这个例子中我们用正确率作为模型预测性能的度量\n",
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "from matplotlib import pyplot\n",
    "import seaborn as sns\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 读取数据 & 数据探索"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>6</td>\n",
       "      <td>148</td>\n",
       "      <td>72</td>\n",
       "      <td>35</td>\n",
       "      <td>0</td>\n",
       "      <td>33.6</td>\n",
       "      <td>0.627</td>\n",
       "      <td>50</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>85</td>\n",
       "      <td>66</td>\n",
       "      <td>29</td>\n",
       "      <td>0</td>\n",
       "      <td>26.6</td>\n",
       "      <td>0.351</td>\n",
       "      <td>31</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>8</td>\n",
       "      <td>183</td>\n",
       "      <td>64</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>23.3</td>\n",
       "      <td>0.672</td>\n",
       "      <td>32</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>89</td>\n",
       "      <td>66</td>\n",
       "      <td>23</td>\n",
       "      <td>94</td>\n",
       "      <td>28.1</td>\n",
       "      <td>0.167</td>\n",
       "      <td>21</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>137</td>\n",
       "      <td>40</td>\n",
       "      <td>35</td>\n",
       "      <td>168</td>\n",
       "      <td>43.1</td>\n",
       "      <td>2.288</td>\n",
       "      <td>33</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Pregnancies  Glucose  BloodPressure  SkinThickness  Insulin   BMI  \\\n",
       "0            6      148             72             35        0  33.6   \n",
       "1            1       85             66             29        0  26.6   \n",
       "2            8      183             64              0        0  23.3   \n",
       "3            1       89             66             23       94  28.1   \n",
       "4            0      137             40             35      168  43.1   \n",
       "\n",
       "   DiabetesPedigreeFunction  Age  Outcome  \n",
       "0                     0.627   50        1  \n",
       "1                     0.351   31        0  \n",
       "2                     0.672   32        1  \n",
       "3                     0.167   21        0  \n",
       "4                     2.288   33        1  "
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 读取数据\n",
    "# path to where the data lies\n",
    "dpath = './data/'\n",
    "data = pd.read_csv(dpath +\"diabetes.csv\")\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数据基本信息\n",
    "样本数目、特征维数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(768, 9)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 768 entries, 0 to 767\n",
      "Data columns (total 9 columns):\n",
      "Pregnancies                 768 non-null int64\n",
      "Glucose                     768 non-null int64\n",
      "BloodPressure               768 non-null int64\n",
      "SkinThickness               768 non-null int64\n",
      "Insulin                     768 non-null int64\n",
      "BMI                         768 non-null float64\n",
      "DiabetesPedigreeFunction    768 non-null float64\n",
      "Age                         768 non-null int64\n",
      "Outcome                     768 non-null int64\n",
      "dtypes: float64(2), int64(7)\n",
      "memory usage: 54.1 KB\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pregnancies                 0\n",
       "Glucose                     0\n",
       "BloodPressure               0\n",
       "SkinThickness               0\n",
       "Insulin                     0\n",
       "BMI                         0\n",
       "DiabetesPedigreeFunction    0\n",
       "Age                         0\n",
       "Outcome                     0\n",
       "dtype: int64"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 查看是否有空值\n",
    "data.isnull().sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Pregnancies</th>\n",
       "      <th>Glucose</th>\n",
       "      <th>BloodPressure</th>\n",
       "      <th>SkinThickness</th>\n",
       "      <th>Insulin</th>\n",
       "      <th>BMI</th>\n",
       "      <th>DiabetesPedigreeFunction</th>\n",
       "      <th>Age</th>\n",
       "      <th>Outcome</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "      <td>768.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>3.845052</td>\n",
       "      <td>120.894531</td>\n",
       "      <td>69.105469</td>\n",
       "      <td>20.536458</td>\n",
       "      <td>79.799479</td>\n",
       "      <td>31.992578</td>\n",
       "      <td>0.471876</td>\n",
       "      <td>33.240885</td>\n",
       "      <td>0.348958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>3.369578</td>\n",
       "      <td>31.972618</td>\n",
       "      <td>19.355807</td>\n",
       "      <td>15.952218</td>\n",
       "      <td>115.244002</td>\n",
       "      <td>7.884160</td>\n",
       "      <td>0.331329</td>\n",
       "      <td>11.760232</td>\n",
       "      <td>0.476951</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.078000</td>\n",
       "      <td>21.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>99.000000</td>\n",
       "      <td>62.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>27.300000</td>\n",
       "      <td>0.243750</td>\n",
       "      <td>24.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>3.000000</td>\n",
       "      <td>117.000000</td>\n",
       "      <td>72.000000</td>\n",
       "      <td>23.000000</td>\n",
       "      <td>30.500000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>0.372500</td>\n",
       "      <td>29.000000</td>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>6.000000</td>\n",
       "      <td>140.250000</td>\n",
       "      <td>80.000000</td>\n",
       "      <td>32.000000</td>\n",
       "      <td>127.250000</td>\n",
       "      <td>36.600000</td>\n",
       "      <td>0.626250</td>\n",
       "      <td>41.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>17.000000</td>\n",
       "      <td>199.000000</td>\n",
       "      <td>122.000000</td>\n",
       "      <td>99.000000</td>\n",
       "      <td>846.000000</td>\n",
       "      <td>67.100000</td>\n",
       "      <td>2.420000</td>\n",
       "      <td>81.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       Pregnancies     Glucose  BloodPressure  SkinThickness     Insulin  \\\n",
       "count   768.000000  768.000000     768.000000     768.000000  768.000000   \n",
       "mean      3.845052  120.894531      69.105469      20.536458   79.799479   \n",
       "std       3.369578   31.972618      19.355807      15.952218  115.244002   \n",
       "min       0.000000    0.000000       0.000000       0.000000    0.000000   \n",
       "25%       1.000000   99.000000      62.000000       0.000000    0.000000   \n",
       "50%       3.000000  117.000000      72.000000      23.000000   30.500000   \n",
       "75%       6.000000  140.250000      80.000000      32.000000  127.250000   \n",
       "max      17.000000  199.000000     122.000000      99.000000  846.000000   \n",
       "\n",
       "              BMI  DiabetesPedigreeFunction         Age     Outcome  \n",
       "count  768.000000                768.000000  768.000000  768.000000  \n",
       "mean    31.992578                  0.471876   33.240885    0.348958  \n",
       "std      7.884160                  0.331329   11.760232    0.476951  \n",
       "min      0.000000                  0.078000   21.000000    0.000000  \n",
       "25%     27.300000                  0.243750   24.000000    0.000000  \n",
       "50%     32.000000                  0.372500   29.000000    0.000000  \n",
       "75%     36.600000                  0.626250   41.000000    1.000000  \n",
       "max     67.100000                  2.420000   81.000000    1.000000  "
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 各属性的统计特性\n",
    "data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "此处得到各属性的样本数目、均值、标准差、最小值、1/4分位数（25%）、中位数（50%）、3/4分位数（75%）、最大值 可初步了解各特征的分布"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEKCAYAAAAIO8L1AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAE+JJREFUeJzt3X+0ZWV93/H3BwbUoHH4MbIow2SwjmnJUpGMFEOrEW0q2jJIxcSyZIKzOsmqURLS1NGkksTYaCJiqC3trGAYLIUgFRmUxkxHwBUTKDMoIKJlpARmYZkB+aUULPDtH+e5cBj23LtnhnPP4d73a62zzt7Pefa53+sa7se9n72fJ1WFJEk72mvcBUiSJpMBIUnqZEBIkjoZEJKkTgaEJKmTASFJ6mRASJI6GRCSpE4GhCSp04JxF7AnDjrooFq6dOm4y5Ck55XNmzffW1WLZur3vA6IpUuXsmnTpnGXIUnPK0n+tk8/LzFJkjoZEJKkTgaEJKmTASFJ6mRASJI6jTQgktyR5OYk30yyqbUdkGRDktva+/6tPUnOSbIlyU1JjhplbZKk6c3GGcSbqurIqlre9tcAG6tqGbCx7QMcDyxrr9XAubNQmyRpJ8ZxiWkFsK5trwNOHGq/oAauBRYmOWQM9UmSGH1AFPCXSTYnWd3aDq6q7wO095e19kOBu4aO3draJEljMOonqY+tqruTvAzYkOQ70/RNR1s9q9MgaFYDLFmyZI8L/NnfumCPv0Nzz+Y/PnXcJUhjN9IziKq6u71vAy4Djgbumbp01N63te5bgcOGDl8M3N3xnWuranlVLV+0aMapRCRJu2lkAZFkvyQvmdoGfgH4FrAeWNm6rQQub9vrgVPb3UzHAA9OXYqSJM2+UV5iOhi4LMnUz/mvVfUXSa4HLkmyCrgTOLn1vxJ4G7AFeAQ4bYS1SZJmMLKAqKrbgdd0tN8HvLmjvYD3jaoeSdKu8UlqSVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnQwISVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnQwISVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdRh4QSfZO8o0kX2r7hye5LsltSf48yb6t/QVtf0v7fOmoa5Mk7dxsnEGcDtw6tP8J4OyqWgbcD6xq7auA+6vqFcDZrZ8kaUxGGhBJFgNvB/607Qc4Dri0dVkHnNi2V7R92udvbv0lSWMw6jOITwP/Bniy7R8IPFBVj7f9rcChbftQ4C6A9vmDrb8kaQxmDIgk+yXZq22/MskJSfbpcdw/BbZV1ebh5o6u1eOz4e9dnWRTkk3bt2+fqQxJ0m7qcwbxNeCFSQ4FNgKnAef3OO5Y4IQkdwAXM7i09GlgYZIFrc9i4O62vRU4DKB9/lLgBzt+aVWtrarlVbV80aJFPcqQJO2OPgGRqnoEOAn491X1DuCImQ6qqg9V1eKqWgr8EvDVqjoFuAp4Z+u2Eri8ba9v+7TPv1pVzzqDkCTNjl4BkeT1wCnAl1vbgmn6z+SDwBlJtjAYYzivtZ8HHNjazwDW7MHPkCTtoT5/6H8d+BBwWVXdkuTlDM4Cequqq4Gr2/btwNEdfR4FTt6V75Ukjc6MAVFV1wDXJNmv7d8OfGDUhUmSxqvPXUyvT/Jt2sNuSV6T5D+OvDJJ0lj1GYP4NPBPgPsAqupG4A2jLEqSNH69HpSrqrt2aHpiBLVIkiZIn0Hqu5L8HFBtYr0P8My5lSRJc1CfM4hfBd7HYCqMrcCRbV+SNIf1uYvpXgbPQEiS5pE+dzGtS7JwaH//JJ8dbVmSpHHrc4np1VX1wNROVd0PvHZ0JUmSJkGfgNgryf5TO0kOYM+m2pAkPQ/0+UN/FvDXSaYW+TkZ+NjoSpIkTYI+g9QXJNkMvInBmg0nVdW3R16ZJGms+l4q+g6D9aMXACRZUlV3jqwqSdLYzRgQSd4PnAncw+AJ6jBY6e3Voy1NkjROfc4gTgd+uqruG3UxkqTJ0ecupruAB0ddiCRpsvQ5g7gduDrJl4HHphqr6lMjq0qSNHZ9AuLO9tq3vSRJ80Cf21x/DyDJflX1o9GXJEmaBK4oJ0nq5IpykqROrignSerkinKSpE6uKCdJ6jTtGUSSvYH3VJUryknSPDPtGURVPQGsmKVaJEkTpM8YxNeTfAb4c+Cp5yCq6oaRVSVJGrs+AfFz7f33h9oKOO65L0eSNClmGoPYCzi3qi6ZpXokSRNipjGIJ4Ffm6VaJEkTpM9trhuS/OskhyU5YOo18sokSWPVZwzive19+NmHAl7+3JcjSZoUfWZzPXw2CpEkTZY+a1Kf2tVeVRfMcNwLga8BL2g/59KqOjPJ4cDFwAHADQwexPtxkhcAFwA/y2BiwF+sqjt24XeRJD2H+oxBvG7o9Y+A3wVO6HHcY8BxVfUaBtNzvDXJMcAngLOrahlwP7Cq9V8F3F9VrwDObv0kSWPS5xLT+4f3k7wU+FyP4wr4Ydvdp72mnp/4F619HYPAOZfBE9u/29ovBT6TJO17JEmzrNd03zt4BFjWp2OSvZN8E9gGbAC+BzxQVY+3LlsZTAJIe78LoH3+IHDgbtQnSXoO9BmDuILB//OHQaAcAfR6cK7N5XRkkoXAZcDf7+o29aOm+Wy4ntXAaoAlS5b0KUOStBv63Ob6yaHtx4G/raqtu/JDquqBJFcDxwALkyxoZwmLgbtbt63AYcDWJAuAlwI/6PiutcBagOXLl3v5SZJGpM8lpjuB66rqmqr6OnBfkqUzHZRkUTtzIMmLgLcwWGjoKuCdrdtK4PK2vb7t0z7/quMPkjQ+fQLi88CTQ/tPtLaZHAJcleQm4HpgQ1V9CfggcEaSLQzGGM5r/c8DDmztZwBr+v0KkqRR6HOJaUFV/Xhqpz2zsO9MB1XVTcBrO9pvB47uaH8UOLlHPZKkWdDnDGJ7kqeee0iyArh3dCVJkiZBnzOIXwUubIsGwWAwufPpaknS3NHnQbnvAcckeTGQqnp49GVJksZtxktMSf5dkoVV9cOqejjJ/kn+YDaKkySNT58xiOOr6oGpnaq6H3jb6EqSJE2CPgGxd5tpFXjqmYYXTNNfkjQH9Bmk/i/AxiR/xmDqi/cymGRPkjSH9Rmk/qP2sNtbWtNHq+oroy1LkjRufc4gAL7B09N1f2N05Uiacufvv2rcJWgCLfnIzbP2s/rcxfQu4H8ymB/pXcB1Sd45/VGSpOe7PmcQvw28rqq2wWASPuB/MFjUR5I0R/W5i2mvqXBo7ut5nCTpeazPGcRfJPkKcFHb/0XgytGVJEmaBH3uYvqtJCcB/5DBqm9rq+qykVcmSRqrXncxVdUXgC+MuBZJ0gRxLEGS1MmAkCR12mlAJNnY3j8xe+VIkibFdGMQhyR5I3BCkosZDFA/papuGGllkqSxmi4gPgKsARYDn9rhswKOG1VRkqTx22lAVNWlwKVJ/m1VfXQWa5IkTYA+z0F8NMkJwBta09VV9aXRliVJGrc+k/X9IXA68O32Or21SZLmsD4Pyr0dOLKqngRIso7BlN8fGmVhkqTx6vscxMKh7ZeOohBJ0mTpcwbxh8A3klzF4FbXN+DZgyTNeX0GqS9KcjXwOgYB8cGq+j+jLkySNF59J+v7PrB+xLVIkiaIczFJkjoZEJKkTtMGRJK9knxrtoqRJE2OaQOiPftwY5Ils1SPJGlC9LnEdAhwS5KNSdZPvWY6KMlhSa5KcmuSW5Kc3toPSLIhyW3tff/WniTnJNmS5KYkR+3ZryZJ2hN97mL6vd387seB36yqG5K8BNicZAPwy8DGqvp4kjUMZoz9IHA8sKy9/gFwbnuXJI3BjGcQVXUNcAewT9u+HphxLYiq+v7UmhFV9TBwK3AosAJY17qtA05s2yuAC2rgWmBhkkN27deRJD1X+kzW9y+BS4H/3JoOBb64Kz8kyVLgtcB1wMHtuYqp5yteNvS9dw0dtrW1SZLGoM8YxPuAY4GHAKrqNp7+oz6jJC8G/hvw61X10HRdO9qq4/tWJ9mUZNP27dv7liFJ2kV9AuKxqvrx1E6SBXT84e6SZB8G4XBhVX2hNd8zdemovW9r7VuBw4YOXwzcveN3VtXaqlpeVcsXLVrUpwxJ0m7oExDXJPkw8KIk/xj4PHDFTAclCXAecGtVDS9Zuh5Y2bZXApcPtZ/a7mY6Bnhw6lKUJGn29bmLaQ2wCrgZ+BXgSuBPexx3LPAe4OYk32xtHwY+DlySZBVwJ3By++xK4G3AFuAR4LSev4MkaQT6zOb6ZFsk6DoGl5a+W1UzXmKqqr+ie1wB4M0d/YvBeIckaQLMGBBJ3g78J+B7DP7gH57kV6rqv4+6OEnS+PS5xHQW8Kaq2gKQ5O8CXwYMCEmaw/oMUm+bCofmdp6+80iSNEft9AwiyUlt85YkVwKXMBiDOJnB09SSpDlsuktM/2xo+x7gjW17O7D/yCqSJE2EnQZEVXmbqSTNY33uYjoceD+wdLh/VZ0wurIkSePW5y6mLzJ4IvoK4MnRliNJmhR9AuLRqjpn5JVIkiZKn4D4kyRnAn8JPDbVOLXWgyRpbuoTEK9iMKfScTx9ianaviRpjuoTEO8AXj485bckae7r8yT1jcDCURciSZosfc4gDga+k+R6njkG4W2ukjSH9QmIM0dehSRp4vRZD+Ka2ShEkjRZ+jxJ/TBPr0G9L7AP8KOq+slRFiZJGq8+ZxAvGd5PciJw9MgqkiRNhD53MT1DVX0Rn4GQpDmvzyWmk4Z29wKW8/QlJ0nSHNXnLqbhdSEeB+4AVoykGknSxOgzBuG6EJI0D0235OhHpjmuquqjI6hHkjQhpjuD+FFH237AKuBAwICQpDlsuiVHz5raTvIS4HTgNOBi4KydHSdJmhumHYNIcgBwBnAKsA44qqrun43CJEnjNd0YxB8DJwFrgVdV1Q9nrSpJ0thN96DcbwJ/B/gd4O4kD7XXw0kemp3yJEnjMt0YxC4/ZS1JmjsMAUlSJwNCktTJgJAkdTIgJEmdRhYQST6bZFuSbw21HZBkQ5Lb2vv+rT1JzkmyJclNSY4aVV2SpH5GeQZxPvDWHdrWABurahmwse0DHA8sa6/VwLkjrEuS1MPIAqKqvgb8YIfmFQyeyKa9nzjUfkENXAssTHLIqGqTJM1stscgDq6q7wO095e19kOBu4b6bW1tz5JkdZJNSTZt3759pMVK0nw2KYPU6WjrXLWuqtZW1fKqWr5o0aIRlyVJ89dsB8Q9U5eO2vu21r4VOGyo32Lg7lmuTZI0ZLYDYj2wsm2vBC4faj+13c10DPDg1KUoSdJ49FmTerckuQj4eeCgJFuBM4GPA5ckWQXcCZzcul8JvA3YAjzCYN0JSdIYjSwgqurdO/nozR19C3jfqGqRJO26SRmkliRNGANCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnQwISVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnQwISVInA0KS1MmAkCR1MiAkSZ0MCElSJwNCktTJgJAkdTIgJEmdDAhJUicDQpLUyYCQJHUyICRJnSYqIJK8Ncl3k2xJsmbc9UjSfDYxAZFkb+A/AMcDRwDvTnLEeKuSpPlrYgICOBrYUlW3V9WPgYuBFWOuSZLmrUkKiEOBu4b2t7Y2SdIYLBh3AUPS0VbP6pSsBla33R8m+e5Iq5pfDgLuHXcRkyCfXDnuEvRM/tuccmbXn8pd9lN9Ok1SQGwFDhvaXwzcvWOnqloLrJ2touaTJJuqavm465B25L/N8ZikS0zXA8uSHJ5kX+CXgPVjrkmS5q2JOYOoqseT/BrwFWBv4LNVdcuYy5KkeWtiAgKgqq4Erhx3HfOYl+40qfy3OQapetY4sCRJEzUGIUmaIAaEnOJEEyvJZ5NsS/KtcdcyHxkQ85xTnGjCnQ+8ddxFzFcGhJziRBOrqr4G/GDcdcxXBoSc4kRSJwNCvaY4kTT/GBDqNcWJpPnHgJBTnEjqZEDMc1X1ODA1xcmtwCVOcaJJkeQi4G+An06yNcmqcdc0n/gktSSpk2cQkqROBoQkqZMBIUnqZEBIkjoZEJKkTgaE5r0ki5NcnuS2JN9L8iftmZDpjvnwbNUnjYsBoXktSYAvAF+sqmXAK4EXAx+b4VADQnOeAaH57jjg0ar6M4CqegL4DeC9Sf5Vks9MdUzypSQ/n+TjwIuSfDPJhe2zU5PclOTGJJ9rbT+VZGNr35hkSWs/P8m5Sa5KcnuSN7Z1D25Ncv7Qz/uFJH+T5IYkn0/y4ln7X0XCgJB+Btg83FBVDwF3spM126tqDfB/q+rIqjolyc8Avw0cV1WvAU5vXT8DXFBVrwYuBM4Z+pr9GYTTbwBXAGe3Wl6V5MgkBwG/A7ylqo4CNgFnPBe/sNRX538A0jwSumev3Vl7l+OAS6vqXoCqmlq/4PXASW37c8AfDR1zRVVVkpuBe6rqZoAktwBLGUyaeATw9cFVMPZlMOWENGsMCM13twD/fLghyU8ymOH2QZ55lv3CnXxH3zAZ7vNYe39yaHtqfwHwBLChqt7d43ulkfASk+a7jcBPJDkVnlqC9SwGS13eDhyZZK8khzFYfW/K/0uyz9B3vCvJge07Dmjtf81gdlyAU4C/2oW6rgWOTfKK9p0/keSVu/rLSXvCgNC8VoPZKt8BnJzkNuB/AY8yuEvp68D/Bm4GPgncMHToWuCmJBe22W8/BlyT5EbgU63PB4DTktwEvIenxyb61LUd+GXgonb8tcDf293fU9odzuYqSerkGYQkqZMBIUnqZEBIkjoZEJKkTgaEJKmTASFJ6mRASJI6GRCSpE7/H6faJrOoDn8hAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Outcome 分布，看看各类样本分布是否均衡\n",
    "sns.countplot(data.Outcome);\n",
    "pyplot.xlabel('Outcome');\n",
    "pyplot.ylabel('Number of occurrences');"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "各类样本不均衡。交叉验证对分类任务缺省的是采用StratifiedKFold，在每折采样时根据各类样本按比例采样"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 训练数据和测试数据分割\n",
    "（随机选择20%的数据作为测试集）"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# data = data.sample(frac=1)  # 对原始数据进行随机排序\n",
    "# test = data.head(154)   # 测试数据20%\n",
    "# train = data.tail(614)  # 训练数据80%\n",
    "\n",
    "# 从原始数据中分离输入特征x和输出y\n",
    "y = data['Outcome'].values\n",
    "X = data.drop('Outcome', axis = 1)\n",
    "\n",
    "#将数据分割训练数据与测试数据\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# 随机采样20%的数据构建测试样本，其余作为训练样本\n",
    "X_train, X_val, y_train, y_val = train_test_split(X, y, random_state=33, test_size=0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据预处理"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 数据标准化\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "\n",
    "# 初始化特征的标准化器\n",
    "ss_X = StandardScaler()\n",
    "\n",
    "# 分别对训练和测试数据的特征进行标准化处理\n",
    "X_train = ss_X.fit_transform(X_train)\n",
    "X_val = ss_X.transform(X_val)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 模型训练"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### default SVC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import LinearSVC\n",
    "\n",
    "SVC1 = LinearSVC().fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classification report for classifier LinearSVC(C=1.0, class_weight=None, dual=True, fit_intercept=True,\n",
      "     intercept_scaling=1, loss='squared_hinge', max_iter=1000,\n",
      "     multi_class='ovr', penalty='l2', random_state=None, tol=0.0001,\n",
      "     verbose=0):\n",
      "             precision    recall  f1-score   support\n",
      "\n",
      "          0       0.75      0.89      0.81        99\n",
      "          1       0.69      0.45      0.55        55\n",
      "\n",
      "avg / total       0.73      0.73      0.72       154\n",
      "\n",
      "\n",
      "Confusion matrix:\n",
      "[[88 11]\n",
      " [30 25]]\n"
     ]
    }
   ],
   "source": [
    "from sklearn import metrics\n",
    "\n",
    "#在校验集上测试，估计模型性能\n",
    "y_predict = SVC1.predict(X_val)\n",
    "\n",
    "print(\"Classification report for classifier %s:\\n%s\\n\"\n",
    "      % (SVC1, metrics.classification_report(y_val, y_predict)))\n",
    "print(\"Confusion matrix:\\n%s\" % metrics.confusion_matrix(y_val, y_predict))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 线性SVM正则参数调优"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "线性SVM LinearSVC的需要调整正则超参数包括C（正则系数，一般在log域（取log后的值）均匀设置候选参数）和正则函数penalty（L2/L1） \n",
    "\n",
    "采用交叉验证，网格搜索步骤与Logistic回归正则参数处理类似，在此略。\n",
    "\n",
    "这里我们用校验集（X_val、y_val）来估计模型性能"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit_grid_point_Linear(C, penalty, X_train, y_train, X_val, y_val):\n",
    "    \n",
    "    # 在训练集是那个利用SVC训练\n",
    "    SVC2 =  LinearSVC( C = C, penalty = penalty, tol = 1e-4)\n",
    "    SVC2 = SVC2.fit(X_train, y_train)\n",
    "    \n",
    "    # 在校验集上返回accuracy\n",
    "    accuracy = SVC2.score(X_val, y_val)\n",
    "    \n",
    "    print(\"accuracy: {}\".format(accuracy))\n",
    "    return accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "No handles with labels found to put in legend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 0.7272727272727273\n",
      "accuracy: 0.7337662337662337\n",
      "accuracy: 0.7402597402597403\n",
      "accuracy: 0.7337662337662337\n",
      "accuracy: 0.7402597402597403\n",
      "accuracy: 0.6883116883116883\n",
      "accuracy: 0.7207792207792207\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzt3XucVHX9x/HXBxBRREUhb6ssyCXACwjiDZGd1NDMS5aClmIGktnF+pWmpWbZr9svK9NVVCxvkLeUlEQTuXhnMRURkJvCeokVBQEFBD6/P74zscKyM7vMmTNn5v18POaxO2fPnPMZLvPZ7+3zNXdHRESkMS3iDkBERIqfkoWIiGSlZCEiIlkpWYiISFZKFiIikpWShYiIZKVkISIiWSlZiIhIVkoWIiKSVau4A8iXDh06eGVlZdxhiIgkyowZM95z947ZziuZZFFZWUlNTU3cYYiIJIqZvZnLeeqGEhGRrJQsREQkKyULERHJqmTGLEREyt0nn3xCbW0ta9as2eJnbdq0oaKigu22265Z11ayEBEpEbW1tbRr147KykrM7L/H3Z1ly5ZRW1tL586dm3VtdUOJiJSINWvWsPvuu38qUQCYGbvvvnuDLY5cRZoszGyImc01s/lmdmkDP7/WzF5KP143s+Wb/XxnM3vLzP4cZZwiIqVi80SR7XiuIksWZtYSuB44AegFDDOzXvXPcfeL3b2Pu/cBrgMe2OwyPwemRBWjFCd3ePRRuPNOWLs27mgk45134IYbYPXquCOROETZshgAzHf3he6+DhgHnNLI+cOAsZknZtYP2AN4LMIYpYhs3Ah//zv07w8nnABf+xp06QJ//CN89FHc0ZWvxYvhW9+Czp3D17/8Je6IJA5RJot9gCX1ntemj23BzDoBnYFJ6ectgP8DftjYDcxspJnVmFlNXV1dXoKWwtuwAcaNg4MPhi99CT78EMaMgcceg27d4HvfCx9Uv/kNrFwZd7TlY8EC+MY3YP/94eab4ZxzYJ994Mkn445MGuPuTTqeqyiTRUMdZFuLdihwn7tvSD+/EJjg7ku2cn64mPtod+/v7v07dsxa2kSKzPr18Ne/Qq9eMGxYSBp33gmzZ8N558Fxx8HkyTB1KvTpA5dcApWV8ItfwIoVcUdfuubMCYmhR4/w9zFqVEgco0eHv5MnnwytQCk+bdq0YdmyZVskhsxsqDZt2jT72lEmi1pg33rPK4C3t3LuUOp1QQFHABeZ2RvA74BzzOxXUQQphbduXfhNtXt3GD4cdtgB7r0XXn0Vzj4bWm02ofvoo2HiRHjuOTjySPjpT6FTp/B12bJY3kJJmjkTzjwzJO/77w8tukWL4LrrYN/0/+RUCt5/H155Jd5YpWEVFRWsXLmSOXPmMHv27P8+5syZw8qVK6moqGj+xd09kgdhDcdCQvdSa+BloHcD5/UA3gBsK9cZDvw52/369evnUtw+/tj9uuvcKyrcwf3QQ93Hj3ffuLFp13nxRffTTw/XaNvW/Yc/dH/33WhiLgc1Ne6nnhr+PNu1c//xj92XLm343CVLwnm//31hY5ToADWew2d6ZC0Ld18PXARMBGYD97j7LDO72sxOrnfqMGBcOmgpQatXw+9/H8Ydvv3t0Cp49FF4/nn44hehqTP6+vaF++4LLZGTT4b/+79w7e99D956K5r3UIqefRZOPDFMKJg8Ga66Ct54A375S9har25FRRhHmjSpgIFKccgloyThoZZF8Vmxwv2Xv3Tv0CH8NppKuT/5ZNNbEtnMnes+fLh7y5burVu7jxrl/sYb+b1Hqdi4MfwdpFLh76RDh/B3tGJF7te44ILQAvnkk8jClAIi7paFlK8PPoCf/SwMRl92GRx6KDz9NDzxBAwe3PSWRDbdu8Ntt8G8eWEM5NZboWtX+PrXYf78/N4rqdzDuM+gQVBVBa+9Flpkb7wBP/4x7Lxz7teqqgqz0mbMiCxcKUJKFpI3dXUhOXTqFLo0jjkGpk+HCRPCwHTUOneGm26ChQvhm9+EsWPDjJ6vfjXMsCpH7vCPf8Dhh8OQISE5XHdd+DP6/vehbdumX3Pw4PBVU2jLi5KFbLN33oEf/CC0JH71q7Cg7uWXNy2wK7SKCvjTn8JMnu9/Hx58EHr3hq98JcRVDjZuDOM6ffuGcZ26ujD1dcECuOiiMAOtufbYAw44QOMW5UbJQpptyZIwYN25c1hlffrpMGsW/O1vcNBBcUcHe+4Jv/3tpq6WiRPDeo1TTgktnlK0fj3cfTcceGBIjh9/HFZcz50LI0ZA69b5uU9VFTz1lMqxlBMlC2myhQth5MiwsvfGG0M3z9y5cPvt0LNn3NFtqUMHuOYaePPNMJYybRoMGBC6ZZ5+Ou7o8uOTT8K4Tc+eYa2KWeiGe+01OPdcaOYWBluVSoVE9MIL+b2uFC8lC8nZ3LlhALl797DyesSIMIB8yy0hcRS79u3hiitCS+NXv4IXX4SBA8NvyZMmhf79pFm7NiTsbt3CgH67dvDAA2HR3NCh0LJlNPc95piQkNQVVT6ULCSrV18N5Th69oR77gldT4sWwfXXh8HspNl551A6ZNGisP5j7lz43OdC4vjnP5ORND76KIzL7L9/GMzfc094+OEwQ+m006BFxP+z27eHQw5RsignShayVS++GAr7HXhg+CD60Y/Cb+XXXgt77x13dNuubVu4+OLQrXb99WEM5sQTQxfVQw8VZ/2jVavCOEznzvDd74Zk8fjjYYHdF76Q/2nJjamqCiVYVBG4PChZyBaeew5OOgn69Qu/Of70p5u6bj7zmbijy782beDCC0OX2s03h9pHp54aZhLdc08ocBi3FSvCuEtlZUjaBx8MU6aEx7HHFjZJZKRSoc7XM88U/t5SeEoW8l9Tp4aqokccERLGL34RBoWvvhp23z3u6KLXunUoyZ0ZrF+7NhTWO+AAuOOOMNOo0JYtC+MsnTrBT34S1ks8+2wo3z5oUOHjqW/gwFD0UV1R5UHJosy5h26MQYPCoOXMmZumm15+OeyyS9wRFl6rVmHjpcw04O22CyW7P/vZsDp83broY1i6dFNJ9p//PIypvPhi6A48/PDo75+Ldu3C6nwtzisPShZlyj188BxxBBx/fOi3/+Mfw6Dv//wP7LRT3BHGr2VLOOMMeOmlsMBw111Dy6Nbt7C96Jo1+b/nW2+FcZTKypC0TzopJPD77w/dYsUmlQprVj78MO5IJGpKFmVm48YwtbJfv1Dx9d13w9TLBQvgO9/ZtpW9papFizCGkSldss8+YXvRLl3CYH8+BnjffDOMm3TpEspxnHFGKFEydmzoBitWqVQY05k2Le5IJGpKFmViw4bwwXPQQWGl9cqVYevSefPgggtg++3jjrD4mYVSJpmiiD16hHIilZXw6183b8vX+fPh/PND4cNbbgkL6F5/Pay67tEj3+8g/444Ioz1aNyi9ClZlLhPPgkfPD17wllnhe6nu+7atHVpvlf2lgOz8Bv1k0+G36j79oVLLw2D0FdfDcuXZ7/G7NlhXKRHj/D3UX/r0i5don8P+bLDDqFIpMYtSp+SRYlauzZ88PToEZLCjjuGrUtnzgxJY/OtS6V5Bg4MNaeefz58f+WVIWlcfjm8996W57/ySuhi6t07dAdefPGWW5cmTSoVxnW0xW1pU7IoMR9/HD54unYN3UsdOsD48fDvf8OXvxz9yt5yNWDApj/n444Lu81VVsIPfxjGhWpqwrjHwQeHXQIvvTTMOPvd72CvveKOfttUVYUW65QpcUciUdJHR4lYtSpsZtOlSxiorqzc9Btvc7Yulebp02fTlq+nnBLKiey3X5hiOmVK2OfjzTcb37o0aQYMCC1XdUWVNnVGJNyHH8Kf/xw+lJYtC10CY8duKvQm8ejdO4xFXHVVKCWy116hhlNTdqRLitat4eijNchd6pQsEur990MhuT/+MQyonnBCKMtxxBFxRyb1desGf/hD3FFEL5UKiwjffTcUNZTSo26ohFm6NGzkU1kZ9mYYPDj0h0+YoEQh8amqCl8nT441DImQkkVCvPPOp+f019+6tF+/uKOTcte3bygNo66o0qVuqCK3eDH85jdhwdb69WHa62WXhTpFIsWiVaswTqZkUbrUsihSCxeGnei6doWbbgoLuDLVUJUopBhVVYWFhYsXxx2JREHJosjMmRNKPnTvHspijxgR/gPefHMyti6V8pVKha+aQlualCyKxMyZYc/kXr3CSuvvfGfTDm777Rd3dCLZHXBAWASqrqjSpDGLmM2YETYZevDBUBb8kktCCYhS3JFOSluLFmF23qRJYUW31vmUFrUsYpLZM7l//9Bsv+KKsLL3f/9XiUKSK5WC2trQdSqlRcmiwDJ7Jh95ZCjFcc01IUn87Gew225xRyeybTLjFuqKKj2RJgszG2Jmc81svpld2sDPrzWzl9KP181sefp4JzObkT4+y8xGRRln1Nw37Zk8eHCoG5TZuvSyy8pz61IpTd27w957K1mUosjGLMysJXA9cBxQC0w3s/Hu/lrmHHe/uN753wYyG0e+Axzp7mvNbCfg1fRr344q3ii4wyOPhD2UX3gh7LD2pz+FrTm1I52UIrMwhfbxxzVuUWqibFkMAOa7+0J3XweMA05p5PxhwFgAd1/n7mvTx7ePOM6827gx7Jl8yCGh4uvSpWGtxIIF8O1vK1FIaUulwr/5117Lfq4kR5QfwvsAS+o9r00f24KZdQI6A5PqHdvXzF5JX+PXDbUqzGykmdWYWU1dXV1eg2+ODRvg7rvhwAPD3hGrV8Ntt4VtMkeO1NalUh40blGaokwWDTVAfSvnDgXuc/cN/z3RfYm7HwR0Bc41sz22uJj7aHfv7+79O8a4OUD9rUvPPjscu/vusHXm8OHaulTKS2VleGhxXmmJMlnUAvU3iqwAtjbmMJR0F9Tm0i2KWcDReY0uD9auDd1L3buHrUvbtg0b38ycCcOGQcuWcUcoEo9UKlSg3bAh66mSEFEmi+lANzPrbGatCQlh/OYnmVkPoD3wbL1jFWa2Q/r79sBRwNwIY22S+luXjhoV1kX84x/w4otw+unaulQklYIPPgiVkaU0RPax5u7rgYuAicBs4B53n2VmV5vZyfVOHQaMc/f6XVQ9gefN7GVgCvA7d58ZVay5WrUq7JncuXMox9G5c5gS+9xzcNJJmvkhkpHZ30LjFqXDPv0ZnVz9+/f3mpqaSK69YkWo0ZTZuvTYY+EnPwklmUWkYZ/9bCh++cgjcUcijTGzGe7eP9t56jBpxPvvw5VXhsG6yy+Hww6DZ54Jc8iVKEQal0rB1KlhAogkn5JFA5YuhUsvhU6d4OqrQ5N6xozwG5K2LhXJTSoVum4javBLgSlZ1PP225u2Lv3Nb0Khv1degQceCAvsRCR3gweHr5pCWxqULAg7e33rW9ClSyjH8ZWvhDUS48aFBXYi0nQdOsBBB2mQu1SU/X4W8+ZB797h++HDQ/dTly6xhiRSMlIpuPHGsCZJFQySrexbFl27hj0k5s+H0aOVKETyqaoK1qwJ08sl2co+WZjBD36grUtFojBoUFikqq6o5Cv7ZCEi0dl1V+jXT8miFChZiEikqqrCrpCrV8cdiWwLJQsRiVQqFRbmPf103JHItlCyEJFIDRwIrVqpKyrplCxEJFJt28LhhytZJJ2ShYhELlMyZ8WKuCOR5lKyEJHIpVJhb/qpU+OORJpLyUJEInf44dCmjbqikkzJQkQi16YNHHmkigommZKFiBREKhW2WX3vvbgjkeZQshCRgkilwtfJk2MNQ5pJyUJECqJ//zCNVl1RyaRkISIFsd12obCgBrmTSclCRAomlYI5c8KulJIsShYiUjBVVeGrxi2SR8lCRAqmT59QtlxdUcmjZCEiBdOyJQwerGSRREoWIlJQqRQsWgRvvBF3JNIUShYiUlCZcQtNoU0WJQsRKajevaFjR3VFJY2ShYgUlFnoipo0CdzjjkZypWQhIgVXVRXWWsybF3ckyffoo6FLL+rEG2myMLMhZjbXzOab2aUN/PxaM3sp/XjdzJanj/cxs2fNbJaZvWJmZ0YZp4gUVqZOlLqito07fP/7cMklocUWpciShZm1BK4HTgB6AcPMrFf9c9z9Ynfv4+59gOuAB9I/+gg4x917A0OAP5jZrlHFKiKF1bUrVFQoWWyrKVNg9mz45jejv1eULYsBwHx3X+ju64BxwCmNnD8MGAvg7q+7+7z0928DS4GOEcYqIgVkFrqiJk8OO+hJ81RXh0WOZxag7yXKZLEPsKTe89r0sS2YWSegM7DF7xlmNgBoDSxo4GcjzazGzGrq6uryErSIFEYqBXV1MGtW3JEk07vvwgMPwPDhsOOO0d8vymTRUA/a1oZghgL3ufuGT13AbC/gDuA8d9/i9w93H+3u/d29f8eOaniIJElmvYW6oprn1lth/XoYNaow98spWZjZ/Wb2BTNrSnKpBfat97wC2FqtyaGku6Dq3XNn4BHgJ+7+XBPuKyIJ0KkT7L+/kkVzbNgAo0eH1lmPHoW5Z64f/tXAWcA8M/uVmX02h9dMB7qZWWcza01ICOM3P8nMegDtgWfrHWsN/B243d3vzTFGEUmYqqowSLthQ/ZzZZMJE2Dx4sIMbGfklCzc/V/ufjZwCPAG8LiZPWNm55nZdlt5zXrgImAiMBu4x91nmdnVZnZyvVOHAePcPzVL+AxgEDC83tTaPk1+dyJS1FIpWLEC/v3vuCNJlupq2GsvOKWxKUN51irXE81sd+CrwNeAfwN3AQOBc4HBDb3G3ScAEzY7dsVmz69q4HV3AnfmGpuIJFP9cYv+/eONJSkWLgwL8X7607D7YKHkOmbxADAN2BH4oruf7O5/c/dvAztFGaCIlK4994SePVVUsCluuglatIARIwp731xbFn929waHodxdvw+ISLOlUvCXv8C6ddC6ddzRFLe1a2HMGPjiF8OixkLKdYC7Z/0V1GbW3swujCgmESkjqRSsXg3Tp8cdSfG77z54773CDmxn5JosRrj78swTd/8AKHAjSERK0THHhBXd6orKrro6TDc+9tjC3zvXZNHCbFOZqnTdJzUYRWSb7b47HHyw1ltkM3MmPP10WITXIoZ64bneciJwj5l9zsxShAV0j0YXloiUk1QKnnkG1qyJO5LiVV0N228P550Xz/1zTRaXEOo2fRP4FvAE8KOoghKR8pJKhcHbZ5/Nfm45WrkS7rgDzjgjtMTikNNsqHRdpur0Q0Qkr44+Glq2DF1RmbUXssldd8GqVfEMbGfkus6im5ndZ2avmdnCzCPq4ESkPOy8c1iUp3GLLbmHLqg+feDww+OLI9duqNsIrYr1QBVwO6EarIhIXqRS8MIL4Tdo2eSZZ+CVV0KrIurd8BqTa7LYwd2fAMzd30yX6EhFF5aIlJuqqlBy+6mn4o6kuFRXQ7t2cNZZ8caRa7JYky5PPs/MLjKz04DPRBiXiJSZo44KtY7UFbVJXR3cey+ccw7sFHNhpVyTxfcIdaG+A/QjFBQ8N6qgRKT87LgjHHGEkkV9t90WyqDEObCdkTVZpBfgneHuq9y91t3Pc/fTtSGRiORbVVUoV/7BB3FHEr+NG0PRwKOPht69444mh2SR3uq0X/0V3CIiUUilwofk1KlxRxK/xx4L5ciLoVUBuVed/TfwkJndC6zOHHT3ByKJSkTK0mGHwQ47hK6oQm7sU4yqq6FjR/jSl+KOJMg1WewGLOPTM6AcULIQkbzZfvsw0F3uRQUXL4aHH4Yf/Sj8mRSDXFdwx1SNRETKTSoFl10GS5fCZ8p0zuXNN4fFeBdcEHckm+SULMzsNkJL4lPc/et5j0hEyloq3X8xeXKohVRu1q0LyeLEE6GyMu5oNsl16uzDwCPpxxPAzoDWWYpI3vXrFxahlWtX1IMPwn/+UzwD2xm5dkPdX/+5mY0F/hVJRCJS1lq1gkGDyne9RXU1dOoEQ4bEHcmnNXcLjW7AfvkMREQkI5WC11+H2tq4Iyms2bND99sFF4QqvMUk16qzK83sw8wD+AdhjwsRkbzLjFuUW1fUjTeGkifnnx93JFvKKVm4ezt337neo/vmXVMiIvly0EGw227llSxWr4a//hVOP704Z4Hl2rI4zcx2qfd8VzM7NbqwRKSctWgBgwfDE0+EKaTlYNw4WLGi+Aa2M3Ids7jS3Vdknrj7cuDKaEISEQldUYsXw6JFcUdSGNXVoQbU0UfHHUnDck0WDZ2X6+pvEZEmy2yvWg5dUdOnw4wZ8W9w1Jhck0WNmf3ezPY3sy5mdi0wI8rARKS89ewJe+xRHlNob7gB2raFr30t7ki2Ltdk8W1gHfA34B7gY+BbUQUlImIWuqImTSrtcYv33w/jFWefHfYiL1a5zoZa7e6Xunv/9OMyd1+d7XVmNsTM5prZfDO7tIGfX2tmL6Ufr5vZ8no/e9TMlpvZw017SyJSKlIpePddmDs37kii89e/wpo1xTuwnZHrbKjHzWzXes/bm9nELK9pCVwPnAD0AoaZWa/657j7xe7ex937ANfx6Sq2vwWKuFEmIlHLjFuUaleUe1hbcfjh0KdP3NE0LtduqA7pGVAAuPsHZN+DewAw390Xuvs6YBzQWIX6YcDYevd4AliZY3wiUoK6dIH99ivdZDFpUlipXuytCsg9WWw0s/+W9zCzShqoQruZfYAl9Z7Xpo9twcw6AZ2BJv2TMLORZlZjZjV1dXVNeamIJEBm3OLJJ8MOeqWmujosPkxCdd1ck8XlwFNmdoeZ3QFMAX6c5TUNTQDbWoIZCtyX3sI1Z+4+OjOO0rFjx6a8VEQSoqoqDALPnBl3JPn19tuhwux550GbNnFHk12uA9yPAv2BuYQZUT8gzIhqTC2wb73nFcDbWzl3KPW6oEREMkp13OKWW2DDhuLa4KgxuQ5wf4Owj8UP0o87gKuyvGw60M3MOptZa0JCGN/AtXsA7YFncw9bRMrFvvtCt26llSzWr4fRo+H448N7S4Jcu6G+CxwKvOnuVUBfoNFBAndfD1wETARmA/e4+ywzu9rMTq536jBgnPunZ1Kb2TTgXuBzZlZrZp/PMVYRKTFVVTB1aviQLQUPPwxvvZWMge2MXEt2rHH3NWaGmW3v7nPSLYJGufsEYMJmx67Y7PlVW3ltkVZIEZFCS6XCb+IvvggDBsQdzba74QaoqICTToo7ktzl2rKoTa+zeBB43MweYuvjDyIieTV4cPhaCl1R8+bB44/DiBFhV8CkyHWA+zR3X55uBfwUuBVQiXIRKYg99oADDiiNooI33RR2wfvGN+KOpGmavK2qu09x9/HphXYiIgVRVQXTpsG6BH/yfPwx3HYbnHoq7L133NE0TXP34BYRKahUKnzYPv983JE03733hjUjSRrYzlCyEJFEOOaYsKI7yV1R1dXQvfumPcaTRMlCRBKhfXvo2ze5g9wvvQTPPQejRhXvBkeNUbIQkcRIpeDZZ+Gjj+KOpOmqq2GHHWD48LgjaR4lCxFJjFQqDHA/80zckTTNihVw110wdGhoISWRkoWIJMbAgWHaadLGLe64A1avTubAdoaShYgkRrt2YQV3ksYt3EMXVL9+cOihcUfTfEoWIpIoqRRMnw4rE7I12rRp8NpryW5VgJKFiCRMKhVKe0+bFnckuamuhl12CeMVSaZkISKJcsQR0Lp1Mrqi/vMfuP9+OPdcaNs27mi2jZKFiCTKDjvAkUcmI1mMGQOffBLWViSdkoWIJE4qFRa5vf9+3JFs3YYNoWjg4MHQs2fc0Ww7JQsRSZyqqjDLaMqUuCPZukcfhTffTP7AdoaShYgkzoABsOOOxd0VVV0Ne+4ZKsyWAiULEUmc1q3h6KOLd3HeokUwYULYs6J167ijyQ8lCxFJpKoqmDUrzDgqNqNHh2KBI0fGHUn+KFmISCJlynwXW+ti7Vq49dawv/a++8YdTf4oWYhIIvXtGxa7Fdu4xQMPQF1d6QxsZyhZiEgitWoFgwYVX8uiuhq6dIHjj487kvxSshCRxEqlYP58WLw47kiCV18NZUguuABalNina4m9HREpJ8U2bnHjjWH203nnxR1J/ilZiEhiHXAAdOhQHMli1Sq4/Xb4ylegY8e4o8k/JQsRSawWLUI5jUmTworuON19dyibfuGF8cYRFSULEUm0VAqWLIEFC+KLwR1uuAEOOihUxS1FShYikmjFMG7x3HPw8sthuqxZfHFESclCRBKte3fYa69411tUV8NOO8HZZ8cXQ9QiTRZmNsTM5prZfDO7tIGfX2tmL6Ufr5vZ8no/O9fM5qUf50YZp4gkl1loXcQ1brFsGdxzD3zta2GP8FIVWbIws5bA9cAJQC9gmJn1qn+Ou1/s7n3cvQ9wHfBA+rW7AVcChwEDgCvNrH1UsYpIsqVSsHRp2Ou60G67LZT4KLUV25uLsmUxAJjv7gvdfR0wDjilkfOHAWPT338eeNzd33f3D4DHgSERxioiCVZVFb4Wetxi48awtuKoo+DAAwt770KLMlnsAyyp97w2fWwLZtYJ6Axkeh1zeq2ZjTSzGjOrqaury0vQIpI8nTtDZWXhxy3+9a8wC6vUWxUQbbJoaE7A1noUhwL3ufuGprzW3Ue7e39379+xFFfBiEjOUimYPDn8tl8o1dVhUeCXv1y4e8YlymRRC9Qv0FsBvL2Vc4eyqQuqqa8VESGVgg8+CFNYC6G2FsaPh/PPh+23L8w94xRlspgOdDOzzmbWmpAQxm9+kpn1ANoDz9Y7PBE43szapwe2j08fExFpUGbcolBdUaNHh9lXF1xQmPvFLbJk4e7rgYsIH/KzgXvcfZaZXW1mJ9c7dRgwzn3TpDd3fx/4OSHhTAeuTh8TEWnQ3ntDjx6FSRaffAK33AJDhoTxknLQKsqLu/sEYMJmx67Y7PlVW3ntGGBMZMGJSMlJpeCOO8KH+XbbRXefhx6Cd96Bm26K7h7FRiu4RaRkVFWF6q8zZkR7n+pq2G8/OPHEaO9TTJQsRKRkDB4cvkbZFTV3brj+yJHQsmV09yk2ShYiUjI6dgyVX6NcnHfjjWFL1/PPj+4exUjJQkRKSlUVPPVUKMGRbx99BH/5C3zpS7Dnnvm/fjE46wefAAAKjElEQVRTshCRkpJKwZo1oWx4vv3tb7B8eXms2N6ckoWIlJRBg8IOelGMW1RXQ8+ecMwx+b92sVOyEJGSsuuu0K9f/sctampg+vTS3uCoMUoWIlJyqqpCN9Tq1fm7ZnU17LgjnHNO/q6ZJEoWIlJyUqmwMO/pp/NzvQ8+gLFj4ayzYJdd8nPNpFGyEJGSM3BgmN6ar66o22+Hjz8uz4HtDCULESk5bdvCYYflZ5DbPaytGDAADjlk26+XVEoWIlKSUqkwKL1ixbZdZ/JkmDOnvFsVoGQhIiUqlQobIU2btm3Xqa6G9u3hzDPzE1dSKVmISEk6/PCwKdG2dEW98w78/e8wfDjssEPeQkskJQsRKUlt2sBRR21bsrj1Vli/HkaNyl9cSaVkISIlK5UK26y+917TX7t+fdiv4thjoXv3/MeWNEoWIlKyMlutTpnS9Nc+8kjYZ7vcB7YzlCxEpGQdemiYRtucrqjq6rBV68knZz+3HChZiEjJ2m67UFiwqcliwQKYOBFGjAiL+0TJQkRKXCoV1km8807ur7npprAL3ogR0cWVNEoWIlLSMuMWuZb+WLMGxowJ3U/77BNdXEmjZCEiJa1Pn1C2PNeuqPvug2XLNLC9OSULESlpLVvC4MG5tyyqq6FrV/jc5yINK3GULESk5FVVwcKF8MYbjZ/38svwzDOhVdFCn46foj8OESl5qVT4mq11UV0dVn4PHx55SImjZCEiJa93b+jYsfFk8eGHcOedoWDgbrsVLrakULIQkZJnFrqiJk0K+1M05M47wzasGthumJKFiJSFVAreegvmzdvyZ+6hC6pv37DJkWxJyUJEykJm3KKhKbRPPw2vvhpaFWaFjSspIk0WZjbEzOaa2Xwzu3Qr55xhZq+Z2Swzu7ve8V+b2avpR5lvOyIi26prV6ioaHjcoroadt4Zzjqr8HElRWRVT8ysJXA9cBxQC0w3s/Hu/lq9c7oBPwaOcvcPzOwz6eNfAA4B+gDbA1PM7J/u/mFU8YpIacuMWzz6aNhBLzM1tq4uLMQbOTIUHZSGRdmyGADMd/eF7r4OGAecstk5I4Dr3f0DAHdfmj7eC5ji7uvdfTXwMjAkwlhFpAykUiE5zJq16diYMbBunTY4yibKZLEPsKTe89r0sfq6A93N7Gkze87MMgnhZeAEM9vRzDoAVcC+m9/AzEaaWY2Z1dTV1UXwFkSklGxeJ2rDhlA0cNCgML1Wti7KZNHQMNHmk9ZaAd2AwcAw4BYz29XdHwMmAM8AY4FngfVbXMx9tLv3d/f+HTt2zGfsIlKCOnWCLl02DXJPnAiLFsGFF8YbVxJEmSxq+XRroAJ4u4FzHnL3T9x9ETCXkDxw92vcvY+7H0dIPA1MeBMRaZpUCiZPDq2K6mrYYw847bS4oyp+USaL6UA3M+tsZq2BocD4zc55kNDFRLq7qTuw0Mxamtnu6eMHAQcBj0UYq4iUiVQKVqyAhx4KW6eefz60bh13VMUvstlQ7r7ezC4CJgItgTHuPsvMrgZq3H18+mfHm9lrwAbgh+6+zMzaANMsTHj+EPiqu2/RDSUi0lSDB4evma6nkSNjCyVRzLe29j1h+vfv7zU1NXGHISIJ0KsXzJ4NJ50E//hH3NHEy8xmuHv/bOdpBbeIlJ3Mam7VgcqdtiIXkbJz0UWwyy7w+c/HHUlyKFmISNn57GfhmmvijiJZ1A0lIiJZKVmIiEhWShYiIpKVkoWIiGSlZCEiIlkpWYiISFZKFiIikpWShYiIZFUytaHMrA54cxsu0QF4L0/hxKlU3gfovRSrUnkvpfI+YNveSyd3z7ohUMkki21lZjW5FNMqdqXyPkDvpViVynsplfcBhXkv6oYSEZGslCxERCQrJYtNRscdQJ6UyvsAvZdiVSrvpVTeBxTgvWjMQkREslLLQkREslKySDOzn5vZK2b2kpk9ZmZ7xx1Tc5nZb81sTvr9/N3Mdo07puYys6+Y2Swz22hmiZu5YmZDzGyumc03s0vjjmdbmNkYM1tqZq/GHcu2MLN9zexJM5ud/rf13bhjai4za2NmL5jZy+n38rPI7qVuqMDMdnb3D9Pffwfo5e6jYg6rWczseGCSu683s18DuPslMYfVLGbWE9gI3AT8j7snZqN1M2sJvA4cB9QC04Fh7v5arIE1k5kNAlYBt7v7AXHH01xmthewl7u/aGbtgBnAqUn8ezEzA9q6+yoz2w54Cviuuz+X73upZZGWSRRpbYHEZlF3f8zd16efPgdUxBnPtnD32e4+N+44mmkAMN/dF7r7OmAccErMMTWbu08F3o87jm3l7u+4+4vp71cCs4F94o2qeTxYlX66XfoRyWeXkkU9ZnaNmS0BzgauiDuePPk68M+4gyhT+wBL6j2vJaEfSqXKzCqBvsDz8UbSfGbW0sxeApYCj7t7JO+lrJKFmf3LzF5t4HEKgLtf7u77AncBF8UbbeOyvZf0OZcD6wnvp2jl8l4Syho4ltgWa6kxs52A+4HvbdazkCjuvsHd+xB6EAaYWSRdhK2iuGixcvdjczz1buAR4MoIw9km2d6LmZ0LnAR8zot8YKoJfy9JUwvsW+95BfB2TLFIPen+/fuBu9z9gbjjyQd3X25mk4EhQN4nIZRVy6IxZtat3tOTgTlxxbKtzGwIcAlwsrt/FHc8ZWw60M3MOptZa2AoMD7mmMpeelD4VmC2u/8+7ni2hZl1zMx2NLMdgGOJ6LNLs6HSzOx+oAdh5s2bwCh3fyveqJrHzOYD2wPL0oeeS/DMrtOA64COwHLgJXf/fLxR5c7MTgT+ALQExrj7NTGH1GxmNhYYTKhw+h/gSne/NdagmsHMBgLTgJmE/+8Al7n7hPiiah4zOwj4K+HfVwvgHne/OpJ7KVmIiEg26oYSEZGslCxERCQrJQsREclKyUJERLJSshARkayULESawMxWZT+r0dffZ2Zd0t/vZGY3mdmCdMXQqWZ2mJm1Tn9fVotmpbgpWYgUiJn1Blq6+8L0oVsIhfm6uXtvYDjQIV108AngzFgCFWmAkoVIM1jw23QNq5lmdmb6eAszuyHdUnjYzCaY2ZfTLzsbeCh93v7AYcBP3H0jQLo67SPpcx9Mny9SFNTMFWmeLwF9gIMJK5qnm9lU4CigEjgQ+Ayh/PWY9GuOAsamv+9NWI2+YSvXfxU4NJLIRZpBLQuR5hkIjE1X/PwPMIXw4T4QuNfdN7r7u8CT9V6zF1CXy8XTSWRdenMekdgpWYg0T0Plxxs7DvAx0Cb9/SzgYDNr7P/g9sCaZsQmkndKFiLNMxU4M73xTEdgEPACYVvL09NjF3sQCu9lzAa6Arj7AqAG+Fm6Cipm1i2zh4eZ7Q7UufsnhXpDIo1RshBpnr8DrwAvA5OAH6W7ne4n7GPxKmHf8OeBFenXPMKnk8c3gD2B+WY2E7iZTftdVAGJq4IqpUtVZ0XyzMx2cvdV6dbBC8BR7v5uer+BJ9PPtzawnbnGA8CPE7z/uJQYzYYSyb+H0xvStAZ+nm5x4O4fm9mVhH24F2/txemNkh5UopBiopaFiIhkpTELERHJSslCRESyUrIQEZGslCxERCQrJQsREclKyUJERLL6fzo2gT6kRMzOAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#需要调优的参数\n",
    "C_s = np.logspace(-3, 3, 7)# logspace(a,b,N)把10的a次方到10的b次方区间分成N份  \n",
    "penalty_s = ['l1','l2']\n",
    "\n",
    "accuracy_s = []\n",
    "for i, oneC in enumerate(C_s):\n",
    "#    for j, penalty in enumerate(penalty_s):\n",
    "    tmp = fit_grid_point_Linear(oneC, penalty_s[1], X_train, y_train, X_val, y_val)\n",
    "    accuracy_s.append(tmp)\n",
    "\n",
    "x_axis = np.log10(C_s)\n",
    "#for j, penalty in enumerate(penalty_s):\n",
    "pyplot.plot(x_axis, np.array(accuracy_s), 'b-')\n",
    "    \n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'log(C)' )                                                                                                      \n",
    "pyplot.ylabel( 'accuracy' )\n",
    "pyplot.savefig('SVM_Diabetes.png' )\n",
    "\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从图中大致可以看出log(C) = -1时，即C = 0.1时，accuracy值最大"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用GridSearchCV对线性SVM进行正则超参数调优\n",
    "在sklearn框架下，不同学习器的参数调整步骤相同： 设置候选参数集合 调用GridSearchCV 调用fit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise',\n",
       "       estimator=SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape='ovr', degree=3, gamma='auto', kernel='linear',\n",
       "  max_iter=-1, probability=True, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False),\n",
       "       fit_params=None, iid=True, n_jobs=1,\n",
       "       param_grid={'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=0)"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "#需要调优的参数\n",
    "# 请尝试将L1正则和L2正则分开，并配合合适的优化求解算法（slover）\n",
    "#tuned_parameters = {'penalty':['l1','l2'],\n",
    "#                   'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]\n",
    "#                   }\n",
    "penaltys = ['l1','l2']\n",
    "Cs = [0.001, 0.01, 0.1, 1, 10, 100, 1000]\n",
    "# tuned_parameters = dict(penalty = penaltys, C = Cs)  #GridSearchCV要求dictionary参数形式\n",
    "tuned_parameters = dict(C = Cs)  #GridSearchCV要求dictionary参数形式\n",
    "\n",
    "SVC_penalty= SVC(kernel='linear', probability=True)\n",
    "grid= GridSearchCV(SVC_penalty, tuned_parameters,cv=5, scoring='neg_log_loss')\n",
    "grid.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Python36\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Python36\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Python36\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Python36\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Python36\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Python36\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Python36\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([0.01521454, 0.01280575, 0.01420884, 0.02521644, 0.11888299,\n",
       "        0.90483699, 8.16097555]),\n",
       " 'mean_score_time': array([0.00119967, 0.00120196, 0.00100183, 0.0012022 , 0.00140467,\n",
       "        0.0012023 , 0.0011992 ]),\n",
       " 'mean_test_score': array([-0.49245193, -0.48213892, -0.48102724, -0.47988287, -0.4799892 ,\n",
       "        -0.48165962, -0.48017849]),\n",
       " 'mean_train_score': array([-0.47978902, -0.46846723, -0.46495812, -0.46535139, -0.46591215,\n",
       "        -0.46556812, -0.46529715]),\n",
       " 'param_C': masked_array(data=[0.001, 0.01, 0.1, 1, 10, 100, 1000],\n",
       "              mask=[False, False, False, False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'C': 0.001},\n",
       "  {'C': 0.01},\n",
       "  {'C': 0.1},\n",
       "  {'C': 1},\n",
       "  {'C': 10},\n",
       "  {'C': 100},\n",
       "  {'C': 1000}],\n",
       " 'rank_test_score': array([7, 6, 4, 1, 2, 5, 3]),\n",
       " 'split0_test_score': array([-0.49176442, -0.4843281 , -0.47678292, -0.47543147, -0.47649277,\n",
       "        -0.47587379, -0.47785398]),\n",
       " 'split0_train_score': array([-0.48416213, -0.47241565, -0.46938579, -0.46935336, -0.4701638 ,\n",
       "        -0.46966042, -0.47116818]),\n",
       " 'split1_test_score': array([-0.4783777 , -0.46522444, -0.4613945 , -0.45835682, -0.45991605,\n",
       "        -0.45978696, -0.45751013]),\n",
       " 'split1_train_score': array([-0.48068719, -0.47237245, -0.4689138 , -0.47057599, -0.47130969,\n",
       "        -0.47123179, -0.46953038]),\n",
       " 'split2_test_score': array([-0.57444622, -0.55658078, -0.56339016, -0.55549193, -0.55260857,\n",
       "        -0.55988794, -0.55567967]),\n",
       " 'split2_train_score': array([-0.46340854, -0.44803385, -0.44439722, -0.44450852, -0.44523109,\n",
       "        -0.44375566, -0.44437745]),\n",
       " 'split3_test_score': array([-0.43255641, -0.44038408, -0.44594593, -0.45129366, -0.45090328,\n",
       "        -0.45276838, -0.44953401]),\n",
       " 'split3_train_score': array([-0.48885782, -0.4787532 , -0.47394136, -0.47421445, -0.47410099,\n",
       "        -0.47528896, -0.47326007]),\n",
       " 'split4_test_score': array([-0.48456946, -0.4636698 , -0.4571781 , -0.45847014, -0.45965195,\n",
       "        -0.45961396, -0.4599197 ]),\n",
       " 'split4_train_score': array([-0.48182943, -0.47076102, -0.46815244, -0.46810464, -0.46875518,\n",
       "        -0.46790377, -0.46814969]),\n",
       " 'std_fit_time': array([2.32096537e-03, 3.99046309e-04, 4.00281219e-04, 7.48558080e-04,\n",
       "        9.79927414e-03, 3.75941136e-02, 5.60308073e-01]),\n",
       " 'std_score_time': array([3.99618284e-04, 4.00257309e-04, 3.37174788e-07, 4.00377040e-04,\n",
       "        4.94258646e-04, 4.00687356e-04, 4.01654247e-04]),\n",
       " 'std_test_score': array([0.04595373, 0.03978194, 0.04239509, 0.03867224, 0.03728132,\n",
       "        0.0398867 , 0.03891162]),\n",
       " 'std_train_score': array([0.0086563 , 0.01057758, 0.01047764, 0.01061953, 0.01048862,\n",
       "        0.01117638, 0.01059781])}"
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# view the complete results (list of named tuples)\n",
    "grid.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.47988287028535365\n",
      "{'C': 1}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid.best_score_)\n",
    "print(grid.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果最佳值在候选参数的边缘，最好再尝试更大的候选参数或更小的候选参数，直到找到拐点。c=1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the correct classification fraction of the zero_one_loss of SVC on test is 0.7532467532467533\n",
      "the correct classification fraction of the zero_one_loss of SVC on train is 0.7736156351791531\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import zero_one_loss  #评价SVC预测模型的性能\n",
    "# 预测\n",
    "y_test_pred_svc = grid.predict(X_val)\n",
    "y_train_pred_svc = grid.predict(X_train)\n",
    "# 使用zero_one_loss评价模型在测试集和训练集上的性能，并输出评估结果\n",
    "#测试集zero_one_loss(y_true, y_pred, normalize=False)\n",
    "print ('the correct classification fraction of the zero_one_loss of SVC on test is', 1 - zero_one_loss(y_val, y_test_pred_svc))\n",
    "#训练集\n",
    "print ('the correct classification fraction of the zero_one_loss of SVC on train is', 1 - zero_one_loss(y_train, y_train_pred_svc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用GridSearchCV发现的最佳参数做预测，测试集上正确分类的比例是0.7532467532467533，训练集上正确分类的比例是0.7736156351791531"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用SVM算法训练出来的模型对糖尿病患者的分类在测试集上的泛化能力稍微好于采用逻辑回归训练出来的模型，\n",
    "由于二者都是线性分类器，所以模型相对简单，错分类误差相对较大。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### RBF核SVM正则参数调优\n",
    "\n",
    "RBF核是SVM最常用的核函数。\n",
    "RBF核SVM 的需要调整正则超参数包括C（正则系数，一般在log域（取log后的值）均匀设置候选参数）和核函数的宽度gamma\n",
    "C越小，决策边界越平滑； \n",
    "gamma越小，决策边界越平滑。\n",
    "\n",
    "采用交叉验证，网格搜索步骤与Logistic回归正则参数处理类似，在此略。\n",
    "\n",
    "这里我们用校验集（X_val、y_val）来估计模型性能"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.svm import SVC"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "def fit_grid_point_RBF(C, gamma, X_train, y_train, X_val, y_val):\n",
    "    \n",
    "    # 在训练集是那个利用SVC训练\n",
    "    SVC3 =  SVC( C = C, kernel='rbf', gamma = gamma)\n",
    "    SVC3 = SVC3.fit(X_train, y_train)\n",
    "    \n",
    "    # 在校验集上返回accuracy\n",
    "    accuracy = SVC3.score(X_val, y_val)\n",
    "    \n",
    "    print(\"accuracy: {}\".format(accuracy))\n",
    "    return accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.7077922077922078\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.7337662337662337\n",
      "accuracy: 0.7402597402597403\n",
      "accuracy: 0.7077922077922078\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.7532467532467533\n",
      "accuracy: 0.7272727272727273\n",
      "accuracy: 0.6883116883116883\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n"
     ]
    }
   ],
   "source": [
    "#需要调优的参数\n",
    "C_s = np.logspace(-3, 1, 5)# logspace(a,b,N)把10的a次方到10的b次方区间分成N份 \n",
    "gamma_s = np.logspace(-2, 2, 5)  \n",
    "\n",
    "accuracy_s = []\n",
    "for i, oneC in enumerate(C_s):\n",
    "    for j, gamma in enumerate(gamma_s):\n",
    "        tmp = fit_grid_point_RBF(oneC, gamma, X_train, y_train, X_val, y_val)\n",
    "        accuracy_s.append(tmp)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从上述部分运行结果来看，gamma参数设置不合适（gamma越大，对应RBF核的sigma越小，决策边界更复杂，可能发生了过拟合） 所以调小gamma值"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6493506493506493\n",
      "accuracy: 0.7337662337662337\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6493506493506493\n",
      "accuracy: 0.7272727272727273\n",
      "accuracy: 0.7532467532467533\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6493506493506493\n",
      "accuracy: 0.7272727272727273\n",
      "accuracy: 0.7597402597402597\n",
      "accuracy: 0.7467532467532467\n",
      "accuracy: 0.6428571428571429\n",
      "accuracy: 0.6493506493506493\n",
      "accuracy: 0.7272727272727273\n",
      "accuracy: 0.7467532467532467\n",
      "accuracy: 0.7467532467532467\n",
      "accuracy: 0.7337662337662337\n",
      "accuracy: 0.6493506493506493\n",
      "accuracy: 0.7272727272727273\n",
      "accuracy: 0.7532467532467533\n",
      "accuracy: 0.7597402597402597\n",
      "accuracy: 0.7402597402597403\n",
      "accuracy: 0.7142857142857143\n",
      "accuracy: 0.7337662337662337\n",
      "accuracy: 0.7467532467532467\n",
      "accuracy: 0.7532467532467533\n",
      "accuracy: 0.7467532467532467\n",
      "accuracy: 0.7337662337662337\n",
      "accuracy: 0.6623376623376623\n"
     ]
    }
   ],
   "source": [
    "#需要调优的参数\n",
    "C_s = np.logspace(-3, 5, 9)# logspace(a,b,N)把10的a次方到10的b次方区间分成N份 \n",
    "gamma_s = np.logspace(-7, -2, 6)  \n",
    "\n",
    "accuracy_s = []\n",
    "for i, oneC in enumerate(C_s):\n",
    "    for j, gamma in enumerate(gamma_s):\n",
    "        tmp = fit_grid_point_RBF(oneC, gamma, X_train, y_train, X_val, y_val)\n",
    "        accuracy_s.append(tmp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYsAAAEKCAYAAADjDHn2AAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzsnXd8Ttcfx983OxKZRCKRJTEiJCKIoGaNKl02FdpSRYeWn1Jdiprdu9VaRf2U0tIqisSesUIlkSGDyEOmjGec3x+R52cksp6lve/XKy+ee84953tvbu73Oef7PZ8jCSGQkZGRkZG5H2bGNkBGRkZGxvSRnYWMjIyMTJXIzkJGRkZGpkpkZyEjIyMjUyWys5CRkZGRqRLZWcjIyMjIVInsLGRkZGRkqkR2FjIyMjIyVSI7CxkZGRmZKrEwtgG6okGDBsLX19fYZsjIyMg8UBw/fjxbCNGwqnr/GGfh6+vLsWPHjG2GjIyMzAOFJEkp1aknT0PJyMjIyFSJ7CxkZGRkZKpEdhYyMjIyMlXyj4lZVIRSqSQtLY3i4mJjmyLzD8LGxgYvLy8sLS2NbYqMjMHQq7OQJKkf8DFgDnwnhFhwV/mHQI9bH+sBbkIIp1tl3sB3QBNAAI8IIZJr0n9aWhr169fH19cXSZLqdC0yMgBCCBQKBWlpafj5+RnbHBkZg6E3ZyFJkjnwOfAwkAYclSRpixAirryOEGLqbfVfBNre1sRKYJ4QYockSfaApqY2FBcXy45CRqdIkoSrqyvXrl0ztikyMgZFnzGLDkCCEOKSEKIUWAc8dp/6I4C1AJIkBQEWQogdAEKIAiHEzdoYITsKGV0jP1My/0b06Sw8gcu3fU67deweJEnyAfyAv24dagbkSJK0UZKkk5IkLb41UpGRkdERyWeyuXzhurHNeCAQQpD/127y9+zh37oVtT6dRUVfvyq7y8OBDUII9a3PFkBXYBrQHvAHxt7TgSRNkCTpmCRJx0x9WqBjx46Ehobi7e1Nw4YNCQ0NJTQ0lOTk5Bq1s3HjRi5cuFDj/rt06UJsbGyNzytnyZIlrFmzptbnG4IhQ4Zw6dKlCstWrlypveehoaFIksTZs2fvqadQKOjVqxeBgYH07duX3NxcfZttFK4k5bLtyzNs+SiWrZ+fIudqrQbu/wqKzp4jZdRo0iZNIm3iC6SOiaL4/Hljm2Vw9Oks0igLTpfjBWRUUnc4t6agbjv35K0pLBXwCxB290lCiG+EEOFCiPCGDatcrW5UDh8+TGxsLHPmzGHYsGHExsYSGxtLTSVKauss6oJSqWTVqlUMGzbMoP3WlIkTJ7J48eIKy8aMGaO95z/88AOBgYEEBwffU2/evHn079+f+Ph4unbtyqJFi/RttsFRlqjZ+UMcdk5WRDzuT3p8DmvnHGb/zwmUFKmMbZ7JoLp2jYw33iB5yBBKk5Nxn/Mu7u+8Q0l8PElPDSbzrbdRXf/3jMz06SyOAoGSJPlJkmRFmUPYcnclSZKaA87AwbvOdZYkqdwD9ATi7j73n8Lvv/9Op06dCAsLY9iwYRQWFgIwffp0goKCaNOmDTNmzCAmJoZt27YxderUWo1Kylm9ejWtW7cmODiYWbNmaY9//fXXNGvWjO7du/Pcc8/xyiuvALBjxw7at2+PuXnZTOChQ4do06YNkZGRTJ8+ndDQUAASExPp2rUrbdu2pV27dhw+fBiAnTt30qNHDwYPHkxgYCCzZ89m5cqVtG/fnjZt2mivY/To0UyePJkePXrQtGlToqOjiYqKokWLFjz77LNaOydMmEB4eDitWrVizpw52uPdu3fnjz/+QK1Wcz/Wrl3LiBEjKizbvHkzUVFRAERFRfHLL7/U5NY+EBz4OYHca0X0jgqiXT9fRr0bQfOO7sTuTOXHtw4Stz8DjebfOdUCoCktRbFsGYn9+pO75Vdcxo2j6fY/cB46FOfhw2i6/Q9cnh5NzsaNJPbth2L5ckRpqbHN1jt6y4YSQqgkSZoCbKcsdfZ7IcQ5SZLmAMeEEOWOYwSwTtw2ESiEUEuSNA3YJZVFE48D39bFnnd/PUdcRl5dmriHoMYOvD2wVZ3ayMrKYsGCBezatYt69eoxb948Pv74Y5599lm2bdvGuXPnkCSJnJwcnJyceOSRRxg8eDCPP/54rfpLS0tj9uzZHDt2DEdHR3r37s1vv/1GSEgICxYs4MSJE9jZ2dG9e3c6dOgAwP79+2nXrp22jXHjxrFixQo6dOjAtGnTtMc9PDzYsWMHNjY2XLhwgaioKK3DOHXqFOfPn8fR0RFfX18mTZrE0aNHWbp0KZ999hlLliwBIDc3l927d/Pzzz8zcOBADh48SIsWLQgLC+Ps2bMEBwezYMECXFxcUKlUWicUFBSEubk5vr6+nD17lpCQkAqvXwjB+vXr+eOPPyosVygUlI9SPT09yczMrNV9NlVSzio4G51OSO8meDZ3BsDO0ZqeY1oS3M2TmJ/i2b3qAmf2pNF1aDMaBzoZ2WLDIYSgYPceri5cgDIlFfvu3XGb8R+s70qRNnd0pNHMmTgNHcrV9xeQtWAhOT+tp9HM17F/6CEjWa9/9LrOQgixDdh217G37vr8TiXn7gDa6M04E+HAgQPExcURGRkJQGlpKV26dMHFxQUzMzPGjx/PgAEDePTRR3XS3+HDh+nZsycNGjQAYOTIkURHR1NcXEzPnj1xdi57gQwePJjU1FQAMjMzadu2LKs5Ozub0tJSrSMZOXIkO3fuBKCkpIQpU6Zw6tQpLCwsSExM1PbbsWNHGjVqBIC/vz99+/YFoHXr1hw8+P9B5cCBA7XHGzduTFBQEABBQUEkJycTHBzM2rVrWbZsGSqVioyMDOLi4rT13NzcyMjIqNRZHDhwAGdnZ1q0aFGt+/VPynwqLlDy18rzuDS2I+Ix/3vK3XwceHJ6GAnHsjiwMYFNS08QEO5G5JMB1HexMYLFhqMkPp6r7y+g8MABrJo2pcm332Lftct9z7Fu2pQm335Dwd69ZC1YyOUJz2P3UFcavf461v733t8HnX/0Cu7bqesIQF8IIejXrx+rVq26p+zYsWPs2LGDdevW8eWXX/Lnn39W2s7tL/Ann3ySt956q8J6lWVy3C/Dw9bWVrsK/n71li5dSpMmTVi9ejVKpRJ7e3ttmbW1tfb/ZmZm2s9mZmaoVKp76t1e5/Z68fHxfPzxxxw5cgQnJydGjx59xwr94uJibG1t2bBhA3PnzgVg+fLl2qmydevWVToFBWjXUDRs2JD09HTc3d0rrfsgIYRgz5q/KS5U8uiLIVhYVpxcKEkSge0b4RvSgJPbUzjxZypJp7IJ6+NN274+WFr9s5IS1Tk5XPvsc26sXYuZnR2NZs3CecRwpGquzpckifrdu2MfGcn1H9eQ/fnnXBr0GC6jRtFg8iTMHRz0fAWGQ9aGMjKRkZHs3btXm8VTWFhIfHw8+fn55OXl8eijj/Lhhx9y8uRJAOrXr09+fv497VhZWWkDuJU5CoCIiAh2796NQqFApVKxbt06unXrRseOHdm9ezc5OTkolUo2btyoPadly5YkJCQA0LBhQywtLbVy8OvWrdPWy83NxcPDA0mSWLFihV5SDPPy8qhfvz4ODg5kZmayffv2O8rj4+Np1aoVgwcP1t6PckehVqvZsGEDw4cPr7T9QYMGsWLFCgBWrFjBY4/db2nQg8PFI1dJPJFFh4F+NGxSv8r6llbmdBjoz6h3I/APacDRrcmsefsQF49e+UekjgqVius//khi337cWLMGp6FDymIRY56utqO4HcnKCtdxY2m6/Q+cnniC6ytXlrX903pEFTG0BwXZWRiZRo0asWzZMoYNG0ZISAiRkZFcvHiR3NxcBgwYQEhICD179uSDDz4AYMSIEcyfP7/WAW4vLy/mzJlD9+7dCQ0NJSIiggEDBuDt7c306dPp0KEDffr0oVWrVjg6OgLwyCOPsHfvXm0b33//PePGjSMyMhIzMzNtvSlTpvDdd98RERFBSkrKHSMDXREWFkZQUBDBwcGMHz+ezp07a8syMjJwdHSkssy43bt307RpU7y9ve84Pm7cOG1a8axZs9i6dSuBgYFER0czffp0nV+Docm/Xkz0uou4+zvSto9Pjc6t72JDn+eCeeK1MGzsLdmxLI5NS06QlaLb+J8hKTx4kKQnnuDqe3OxbtECv02b8Hj7bSxuTcHWBQtXVzzem4PfzxuwbtqUK2+/TdJTgyk8fEQHlhsZIcQ/4qddu3bibuLi4u45JlM5+fn5QgghSktLRf/+/cWWLVu0ZQMHDhSJiYl31BNCiLlz54pXX33VsIZWwqJFi8Ty5csN0teD8mxp1Bqx6YMT4quX9oicrMI6taVWa8S5feli2bRo8dnEXWLnijhRkFOsI0v1T0lKikidPFnENW8h4nv1Frl//ik0Go3e+tNoNCL399/FxR49RFzzFuLySy+LkstpeuuvtlCWcFTlO/ZfE7OQqZo333yTPXv2UFxcTL9+/e4Iqi9cuJCMjAz8/f3ZsmULixYtQqVS4evry/Lly41n9G24uroyevRoY5thUpzenUb63zfoMboFjg3r1aktMzOJoM6NCQhz49i2ZE79dZnEE1mE9/clpGcTzC1Nc6JCXVCI4uuvuL58BVha0vDVV3GJGoOZHka+tyNJEg79+mHfvTuK779H8e13FOzejcuzz9Bg/HjM6tXt92FoJPEPmH8ECA8PF3dvq3r+/HlatmxpJItk/slU99kSQnDx0H5UpSX4hoRh51T3qY7qcj2jkPXzj9IkyIVHXmit88yunKs32f9zAsmns3FoaEvnpwLwC2lgMhlkQqMh95fNZH34Aepr2Tg+/jgNp07FspFbrdo7mHGQI1dqP51krSggcN1hGu+Pp9jFjvjhHcmMDASzut8v93ruDGtRu0WzkiQdF0KEV1VPHlnIyOgJtUrFru+/5Myu/wfhG/kH4tc2HP+24TRqGoCZmX6yi9QqDTuXx2FpY06P0S308gJ3alSPAZPakBqnYN/6eH7/6gxeLZzpMjQQ18b2VTegR26eOMnV+fMpPnsW25AQGn3+ObZtap+JfyDjAJN3TkYg6nYvu0NggBVj/rxJ6y/+wmbzblY+bEmiZ91GZa0btK61s6gu8shCRqYWVPVsldy8yW8fLSD51Ak6PjGMwA6dSIo9TtLJY2TG/40QGmzrO+Ab2g6/tuH4tmmLbX3dpVke3nKJY9uS6f98a/zb6l8KR63WcC46nSO/JlFarCb4IU86DPTDxs6wG0Qpr1wha8lS8n77DQs3N9ymT8NhwAAks9q/jC/euEjU71F42Huwst9K7K3q7giFRkPu5i1kfbBUJ6OeulDdkYXsLGRkasH9nq18RTabFrxDdloqD4+fQuuefe4oL8rPI/n0SZJOHiM59jhF+XlIkhnugc3wDw3Hr204br7+tX7BXbmUy8bFx2ke4U6vqKBatVFbigpKOfJrEuei07GqZ0HHgf606toYM3P9xjM0xcXauABqdVlc4LnnMLOzq1O7WTezGLVtFGqNmjUD1uBup9t1N2XxlK+5vnw5WFrS4PnncRkbpfd4yu3IzgLZWcjoj8qerazkS2xa8A6lxcUMfHUmvm3aVnD2/9Fo1FxNTODSyWMkxx7jSmI8AHZOzviGtsO/bTg+bdpiXa96Lz1liZqf5h5BoxYMf7MDVrbGmWlWpBcQsz6e9L9v4NLYji5DAmnS0kXn/QghyN++nauLFqHKyKR+3764TZ+OlVeFuyHUiJvKm4z9YyzJecms6LeClq76e5eUpqZyddEiCnbuwtLLC7cZ/6F+794Gif/IzgLTchYdO3akpKSE69evU1RUhKdn2cP8yy+/1Eh5duPGjQQFBVVbrqKcLl268Nlnn2kXqNWUJUuW0LhxY0aOHFmr8w3BkCFDWLhwIf6VSC3ExsYyceJE8vPzMTMz48SJE/fso61QKBg6dCipqan4+/uzfv167TqS26no2Uo6eYxfP1qIjZ09T7z+Ng29fWt8DYU5N0g+daJs1HH6BCWFhZiZm9O4eUv8QstiHa5NfCp9iexZ8zfnYtJ5fGpbPJsZLpheEUIIkk5ls39DPHnZxfiFNCDyqQCc3HSTBVQcF8eV+fMpOnYc6xYtaDRrJna3VAzqikqj4uXdL7MvfR+f9vyUh7wMo/lUeOAAV99/n5L4BOpFRNBo5kxsmjfTa5/VdRZGXx+hq58HZZ3FDz/8ICZPnlzr80eNGiU2bdpU4/M6d+4sTp48Was+S0tLRZs2bYRKparV+YZi586dYuLEiRWWlZaWiuDgYHH69GkhhBDXrl0TarX6nnpTp04VixcvFkII8d5774lZs2ZV2N7dz1bsn1vF0uEDxcoZL4l8RXZdLkOLWqUSl8+fFdFrlosV06eIJUMHiCVDB4ivXxgr/vzmUxF/9JAoKbqprZ90+pr47PldYt+GeJ30ryuUpSpx7Pck8fVLe8QXk/8S+3+OFyU3lbVvLztbZMx+U8S1aCn+jugkrq/7SWh0+GxqNBrx3sH3RPDyYLHu/DqdtVvt/pVKofjxR/F3h44irmWQyHjnHaG8fl1v/VHNdRZGf8nr6udBdhbbtm0TERERom3btmLo0KGioKBACCHEtGnTRMuWLUXr1q3Ff/7zHxEdHS2cnZ2Fr6+vCAkJEUlJSdXu93ZnsWrVKhEcHCxatWolZs6cqa3z1VdficDAQNGtWzfx7LPPipdfflkIIcTWrVvFs88+q6138OBB0bp1a9GpUycxbdo0ERISIoQQIiEhQXTp0kWEhoaKsLAwcejQISGEEDt27BDdu3cXTz31lAgICBBvvPGGWLFihQgPDxetW7fWXseoUaPEpEmTRPfu3YW/v7/Yu3evGDNmjGjevLl45plntP2PHz9etGvXTgQFBYl3331Xe1ylUglfX98KndrmzZtFVFRUlffJ399fZGVlCSGESE1NFUFBQRXWK3+2NGq12LNqmVgydIDYuOCdO17euiZPcU2c3rVdbF4yT3wSNVgsGTpAfDjyMbH+vTfEwZ83iG9f+UWsefegUJXe6wRNgYKcYrFz+Tnx2fO7xLLpMSJuf7rQqKu/KE5TUiKyv/9BXGgXLuJaBYsr898Xqtxcndu5/OxyEbw8WCw5ukTnbdcE1Y0bIvO9uSIuqJW40L6DUKxYKTSlpTrvp7rO4t+TOvv763DljG7bdG8N/RfUqQlZotwwEuUXL15ECEGfPn3Izs5m1KhRvPbaa/fcn5pIlCtLS/jj8w+5eGgfIX0G0HPsBMzM9Se0V9+lAa179qF1zz6oVUrSL5wnKfYYSSePsv+nHwBQl7ixZ1Us/m3D8WrVGksrwwVKq8LO0ZpeUUEEd/Ni3/qL/LXyAmf2pNN1aCAeAZVLoQshtMqupcnJ2HV7iEYzXsfa36/Sc2rLzpSdLD22lId9HmZqu6k6b78mmDs54T77DZyHlUmhX50/nxs//USj11+vUhFXH/x7nIWJIkuUG0aiXKVSsX//fg4fPoyNjQ09evQgPDycbt263fd+VRYb0Gg0/Pe9N8iM/5tuTz9LuwGPG3QxmrmFJd7BbfAOboN7QH/+XHYI31YFqEqTOLtnB7Hbf8PC0oomwW3waxuOX2g4To1MQ0G3ka8DT05vR/zRqxzYmMjGJScIbN+ITk80vUcKvSQxkasLFlIYE4OVnx9Nvv4K+yp+Z7Xl9LXTvB7zOq0btmZ+l/mYSaaxIt06MJAmy77T7rVxefx47Lt3p9HrM7Cq4U6bdeHf4yzqOALQF0LIEuWGkCj38vKie/fuuLq6AtC/f39OnDhxj7OojkS5qrSUwpwbXEu6xKCpMwnsGFnpPdE35SKBns29GfBSGGZmEqrSUtLOnyXp5DGSYo/x1/dfAeDc2Av/tu3wC22PZ8tWWNRCXVVXSJJEsw7u+IU05MT2FE7uSCUp9hph/XwIfdgbs6ICrn3+OTfWrMXM1pZGM1/HeeTIWinCVofL+Zd58a8XaWjbkE96fIKNhWnt3yFJEvV79sCuS2durFpF9hdfkjhwEC5PP02DFyZiXr9qJeG68u9xFiZKZGQkL7/8MpcuXcLf35/CwkIyMjJwd3enuLiYRx99lI4dO2q/OVclUV4VERERTJ8+HYVCgaOjI+vWrWPatGm0bt2aGTNmkJOTg52dHRs3biQ8vCxBojKJ8vDw8HskygMCAgwuUd6vXz9teblEecOGDRk8eLD2eJMmTfjwww8pKirC0tKS6OhoXn/99XvaL5conzZtWoUS5aVFReRczQQhGPr2+3gENtf5NVYXoRHsWhGHENB7bBBmt2QjLKys8A0JwzckjB5M4EZmunZBYOyf2zi+dTOW1jZ4tw4luMfDNG3XwWgSHZbW5nQc5E/Lzh4c3JjIkV+TOLsjkSYJW3HJOI7HU0/R8OWXsHDRfdptObkluUzaOQmVRsUXvb/A1dZVb33VFTMrK1yffRbHQYPI+ugjrv/wA7k//0TDwV1wmvahXn+PsrMwMrdLlJfe2sd3/vz52Nra8uSTT1JSUoJGo7lDovz5559n6dKlNU67hTslyoUQDBw4kAEDBgBoJco9PT3vkSi/fQ/scony+vXr89BDD90hUT548GDWrl1L79699S5R7u/vX22JcldXV1566SXatWuHmZkZAwcO1E6FjRs3jpdffpnQ0FBmzZrF0KFD+frrr/Hz8+Onn37StlGUn0/etauYW1pi5+RsVEcBcOqvy6T/nUOPp1vg0MC20nrOHp44e3gS1n8QyuJiLsed4dLJY1w6foTNxw7hHRxCj6jxNKhFqq+ucHC1pc+zQXhmxHDioi1/ew0Er4E4S/Xw+UuBTzB4BDhhbqHbqaFSdSmv7H6F9IJ0vnn4G/wcdR8H0QcWDRvSeO5cnFvC1a/Wkr8rGqdX1WCuv1e6vM5CRktBQQH29vYolUoee+wxXnjhBW0MYdCgQXz00Uf4+/tr6wHMmzeP69evs3TpUmOaDsDixYtxc3MjKipKp+0KISjMuUHBdQVWtrY4NfLg74sXjfpsKTIK+O/8Y3USCdSo1ZzasY0D63+k5OZNQvr0J3LIKJ3KjlTbluJiMma8Tv727TiNGIHNc6+QeiGXlLPZpMfnoFEJLK3N8WrhjE+wKz7Brtg7122qSAjBzH0z2XppKwu6LmCA/wAdXY0BUKtg2zQ4/gMi6Ak0vRZj7lo7WRdZSFCmxsgS5fcihCDvWhZF+XnY2tfHoaFbnXSGdIFapWHnD3FY2dZNJNDM3Jy2/QbSonM3Dvx3Dad2bOPCvr10GjKKkIf7Y25hmNeD6vp10l6YRNHp07jNmIHL2CgkScLZ04GQXk0oLVaR/vcNUs4qSDmrIOlUNgCunvZax+Hu71BjSZHPYz9n66WtvNj2xQfLUZTkw3/HQsJO6DIVqedbmBvgmZRHFjIylaBRq8nJukLpzZvYO7tg5+yifTEb89k69Esix/9Iof/E1viH6k4kMDs1md0rviX17ClcvbzpHjW+SrmSulJyKYnLzz+PKiuLxosX4dCnz33rCyG4nllIylkFqWcVZCbkotEIrGwt8A5ywbuVK96tXLBzvP8U6C8Jv/Dm/jd5IuAJ3o1812Rk1askNx3WDIOsOHj0A2g3ts5NyiMLGZk6oFYquXElA7VSiaNbI6NMzVREZmIuJ7an0DLSQ6eOAqCBty+DZ88l8dhh9qz6jp/nvUnT8I50e/pZnN0b67QvgJtHj3J5yotI5ub4rFyB7V3pzhUhSRKuje1xbWxPWB8fSopUpJ2/rh11JBzPAqChd33tqMPN10Eb/Ac4lHmIdw+8S4RHBG92evPBcRSZp2HNUCgpgFHrIaC3QbuXRxYyMnehLCnmxpVMhEaDUyMPrCvY0cwYz1ZpsYqf5h5BCBg+W78igSqlkhPbNnNo40+olUraDXiMjk8Mq/Be1IbcX38jc9YsLL28aPLN11g1aVLnNoUQZF8uKBt1nFNw5VIuQoCNnSXerVzwCXZF2TiH56LH4W7nzsr+K6lvpf+UU50Qv6Ns6snGEUauB/dgnTUtjyxkZGpBcWEhuVlXMDMzw9nTy6RWQO//OYE8RTFPvBqmdzVZC0tLOjw2mFbdehGzdgVHt/zMub276DJiDMHdetc6biOEQPH111z76GPqtW+P16efYO5U+ertmiBJEg2969PQuz7hj/hSXKjkctytUcc5BRePXEWgob/DRLpGhlF8Bey9BJIOdqrTK8e+h63ToFFQmaNw0P0orzrIIwsZmVvczM0hL/saltbWOLk3vm+A19DPVvLpbLZ+cZq2fbyJfDLAYP2WcyUxnt3LvyHj4nka+QfQI2oCni1qtleGUCrJfOcdcn/eiMPAgXjMm4uZlZWeLL6TgpJCXlk/A+myA13V/clPVwJg62CFTysXfIIb0KSlM9b1jLdQ8R40Gtj1Duz/GAL7wODvwVr3I6Hqjiz0GkKXJKmfJEl/S5KUIEnSPSugJEn6UJKk2Fs/FyVJyrmr3EGSpHRJkj7Tp52GoGPHjoSGhuLt7U3Dhg0JDQ0lNDSU5OTkGrWzceNGLly4UOP+u3TpUq1Fe5WxZMkS1qxZU+vzDcGQIUO4dOlShWUJCQnY2tpq7/vkyZO1ZUII8rOvkZd9jcKSEoZGjaNFy5b07duX3NxcQ5lfKUX5pfy1+gKunvZ0HFix/Lq+cW8ayPA5i3jkpekU5uaw7u3/8NvHi8jLzqrW+er8fC4//zy5P2+kwaQXaLxoocEchVqjZua+1zkqYhj9dF/GvNmVcYu60GtsSzybOZF0Kpvt355l2bR9bFp6ghPbU1CkF+hlUWm1URbBhnFljiL8GRi+Vi+OoibobSwrSZI58DnwMJAGHJUkaYsQIq68jhBi6m31XwTuTr14D9irLxsNSbmg3vLlyzl27BiffVY7/7dx40bMzMxqvJ9FXVAqlaxatYoTJ04YrM/aMHHiRBYvXsyXX35ZYXnz5s3vcZgajYa8rKsUFxZQz9GJhfPm079/f6ZNm8bcuXNZtGgR8+bNM4T5FSKEYM+Pf1NyU8mgl0IxtzRe2q4kSbTs3I2Adh05suVnjm35mcRjh2k/6CnaD3oSS+uK1z0oMzK4/PxESpLNr/WxAAAgAElEQVSS8Jg/H6cnnzCo3YuOLmJP2h5mdZxFtyZl8i71HKxoEeFBiwgPNGoNV5PytNNVBzclcnBTIvbO1ni3KguSe7VwxsrGQLP2hdmwdgSkHYGH34PIF8EEgvD6fPI6AAlCiEtCiFJgHfDYfeqPANaWf5AkqR3QCKhcEOkfwu+//06nTp0ICwtj2LBhFBYWAmUrqoOCgmjTpg0zZswgJiaGbdu2MXXq1FqNSspZvXo1rVu3Jjg4mFmzZmmPf/311zRr1ozu3bvz3HPP8corrwCwY8cO2rdvj/ktRdVDhw7Rpk0bIiMjmT59unZDpcTERLp27Urbtm1p166d1kHu3LlTqw4bGBjI7NmzWblyJe3bt6dNmzba6xg9ejSTJ0+mR48eNG3alOjoaKKiomjRosUdK8gnTJhAeHg4rVq1Ys6cOdrj3bt3548//kCtVlfrPqhVKm5kplNcWED9Bg1xaNCQLVu2aBf1RUVF8csvv9TqHuuKvw9d4VLsNToO8qeBV933ftYFljY2dB46inEffoV/uw4c3LCGH6a+wIX9e+/5Nl507hxJw4ahzMzE+9tvDO4oVsetZs2FNYwJGsOIFiMqrGNmboZHgBMRjzdl2BsdiHq/Mz2eboGbrwPxx67y+1dnWPZaDJs/OknszlRuXCnU36gjOwG+6w1XTsOQFdD5JZNwFKDfALcncPm2z2lAx4oqSpLkA/gBf936bAYsBZ4GeunCmIVHFnLhes2nb+5HC5cWzOgwo05tyBLlhpEoh7KpqLZt2+Lo6Micd9+lpZ8PGpUKJ3cPbOzKXsQ1kSjXN3mKIqJ/uohHgCOhvb2NZkdlODR0Y+ArM0jrO4Ddy79l6yeLObl9Kz3HTqCRfwD5u3eT/to0zJ0c8fn+e6wDAw1q367UXSw6uohe3r14LfxeOfrKsHe2JqhzY4I6N0at0nAlMZeUswqSzyrYvyGB/RsScGhgg08rV7yDXfFs7oyllQ6k6VMOwroRIJlB1K/QRDe7/ukKfTqLitxhZe54OLBBCFH+lXASsE0Icfl+OdCSJE0AJgB4e5veH1N1kCXKDSNR7uXlRWpqKi4uLuyPjmbo8GHs2/Ennk0DsbKpXDbCWDn4QiPYtfw8cKdIoCni1TKYUe9/wNndO9m3biWrZ00lsIkf3tt34xjYnCZffYlFBXpd+uTMtTO8Hv06wQ2Ceb/r+7WWGze3MMOzuTOezZ2JfCqAPEURqefKMqzOH8zkzN50zC3N8GxWLkPigmPDWqQXn9kAv7wATt4w6r/gYpzY1P3Qp7NIA25PnvYCMiqpOxyYfNvnTkBXSZImAfaAlSRJBUKIO4LkQohvgG+gLBvqfsbUdQSgL2SJcsNIlIeGhmJjY0NRfh4BXo3x8vRCcbMYv7scRXUkyg1B7K7LZMTn0HPM/UUCTQUzM3Pa9OpLs46R/DVzGheSE0lq6UunoYNpoqPU2OqSlp/GlL+m4Grryic9P8HWQnf3z8HVluCHPAl+yBOVUk1GfM6tdR3XifnpIjE/gVOjevjcinU0DnS6f5xJCNj3AeyaA96RMPxHqKc/hd26oE9ncRQIlCTJD0inzCGMvLuSJEnNAWdA+/VSCDHqtvKxQPjdjuKfgixRXjNqK1GelZWFtRkU5eaSduUKl9PTCWzW7J72q5IoNwSK9AIObU7EL6QBLTp5GLz/2qIpKiJ75hv47YgmYMhTnLOzJGb9as5E76L7mOfwD9O/FHpuSS6Td01GqVHyQ68faGDbQG99WVia4x3kindQmaR5TtZNUs+VrSQ/G53Oqb8uY2Ftjlfz/4sf3rG5k1oJv02Fk6ug9RB47HOwMJ11PXejN2chhFBJkjQF2A6YA98LIc5JkjSHsj1ft9yqOgJYJ/4pCz5qiCxRXjNqI1EuhIZtm3/h/UWLsbK2xsramm+//VZrd3Ulyg2BWqlhxw9xWNta1Ekk0NCosrO5PGkyxWfO0GjWTFzGjKEZkBR7nD0rvuWXRe/h06Yt3cc8R4MmPnqxQalWMnXPVFLzU/nm4W/wdzLsVI6TWz2c3OrRpkcTlKXqO8QPk0+XiR+6NLYrcxyBNrgffwnz5F3w0HTo8YbJBLIrQ16UJ6PlnyhRrlGrybmaSWlR0T1igHVBX8/WwU2JnNiewiOT2uDXRn/finVJyaVLXJ7wPKrsbDyXLKZ+7zs1i9QqVZkU+n9/pLSoiNA+A+g0ZCS29rpbNyCE4I19b/DrpV+Z32U+A5sO1FnbdUUIwY0r/x91ZMTfQKMGK+kmTXwlvLu0xSfYtUrxQ30hy33I1Jh/mkS5qYoBVkZGQg4n/0whqLPHA+MoCo8cIW3Ki0iWlvisWolt69b31DG3sCCs/6AyKfT1PxK7fSvn9+0hcugoQnr3x8y87plEX536il8v/crk0Mkm5SigLEnCxcMOFw87QoOuU7r6FdLy/UjxeJmUVGsSV5VlaTZoUi653oBGfg4ml9Qgjyxk/pEoi4u5cSUDhMDR3QNrW90I4JWj62erXCQQYNjsDoZbAFYHcrdsIeON2Vh5e9Pk66+w8vKq1nnXUpLYveJbLp87jauXNz3GTsCndWit7dicsJnZ+2fzWNPHeK/ze6Y7dXdxO/x3HNg6l2U8NQoqk1zPKNROV2Um5iI0Ams7C7xblokferdyxba+/la7V3dkITsLmX8cxYUFZWKA5hY4u3tgoQcxQF0/W7tXnSfuQCZPvBZG4wDDZg/VFCEE2V9+SfYnn1KvY0e8PvkY81vxn5q0kXDkIHtXLyM36yoB7SPoNvpZnNxrFtA/nHmYiTsm0q5RO77s/SWW5iak7XQ7R76F3/8D7q3LxADrV5xlV3JTyeXzN0g5m03KuesU5ZWCBG4+Dv+XXPeur1PxQ9lZIDuLfyOFuTnkZ1/D0toGJ3cPve32pstnK+l0Ntu+OE1YXx86PdFUJ23qC1FaSubb75C7aROOjz2Gx3tzkOqg8aQqLeX41l84vGk9GrWKsAGPE/HEUKyqMRJMzEnk6W1P41bPjZWPrMTBygSnGTUa2PkWHPgUmvWDp5aBdfVW4guN4Nrl/P9LriflgQDb+pZaGZImLV2wsaubg5RjFjL/OvIV2RTm3MDazg5HN3fMjLz9aXUoyi9l96rzuHrZ02Ggn7HNuS/qvDzSXnqZm4cO0WDKFBpMnlTnKR8LKys6PjH0/1LomzcQt3cXPZ+ZSLOOnSs9L7som0k7J2FlbsUXvb8wTUehLIKNE+D8FugwAfotALPqx2ckMwk3HwfcfBxoP8CPooLS/0uun1Hw96ErSBKYNbQBL1smTtDvroays5D5R6BRqynMuVG2T7ZbI9Odt74NIQS7V1+gpEjFY6+0xdzCdJ2bMj2d1OefpzQlFY8F7+NUS7mZyrB3caX/5FcJ7TOAXd9/xa8fvE+X4WPo8PiQe36XN5U3mbJrCjdKbvBD3x9obG+c/R3uS8G1MumOtGPQ932IeKHOqbG29lY06+BOsw7uaDSCrOQy8cMdu5KRSpQ6MrxyTPfp/IchS5Trl9LiYsZPeYlMheK+jiIpKQk7Ozs++uijCssTExPp0KEDAQEBjBw5EqVSf3+EFw5eIelUNhGPNcXV0zREAiui6MxZkoYNR3U1C+9vv9W5o7gdj8DmDJ+ziBadu7Fv3Up2fPMp6ttW+Ks1al6PeZ04RRwLuy6kVYNWerOl1mTHw3e94MpZGLYKOk3S+RoKMzMJd39HfLo15jvbIhr099Rp+xX2qfceZIAyTabY2FjmzJnDsGHDiI2NJTY2tsaL6mrrLOpCuUT5sGHDDNpvTVAW3yRq5Ag++uTT+9Z79dVX6d+/f6Xl06dP5z//+Q8JCQnUq1dPb2nBedlFxKy/SONAJ0J71X1LUX2R/9dfpIwZg5m1Nb7r1mIXUaEWqE6xsLTkkRenEfHkMM789SebFr5Lyc2bACw5toTdl3czo8MMenj30LstNSZ5f5lqrPImjN0KLfWbxrs/oWyxX5dm+tfekp2FCSBLlJddR10kykuLiunWvRvbt2+vVKJ8w4YNtGjRotK9QNRqNdHR0TzxRJmMtr4kyjUawa4VZSKBvca2NNltPa+vXEXa5ClYBwTg+9M6rJsaLvguSRKdhz1Nn4kvcfncada9/R9WHvqG1edXM7rlaEa1HFV1I4bm9HpY9TjYu8FzO8GrXdXn1JH9Cdm42lnR0l3/MZt/Tcziyvz5lJzX7Tdy65YtcL/tZVsbZInyukuUP/nEEzSoZ4O9s0ulEuX5+fksXbqUnTt38v7771d4b65du0aDBg20TtHLy4v09PRa3ef7cWpnmUhgr6iWOLiankigUKu5unAhN1auwr53LzwXL8bM1jh2tu7RBwdXNzYunUPBFwn0eTSSaeHTqj7RkAgB0Utg91zw7Vo29WTrbIBuBfsSsokMaGCQBXzyyMLI3C5RHhoayo8//khycvIdEuWbNm3Czs5OJ/3dLlFuaWmplSgvP+7s7IyVldUdInyZmZlavaWKJMrLKSkp4dlnnyU4OJjhw4cTF6fdFFErUW5jY3OPRPntI6SKJMrNzMy0EuUAa9euJSwsjLCwMM6fP8+Z06cAsLS11UqU382bb77J9OnT73sfK0oj13WgPDutgENbEvEPbUjzCOMo2t4Pzc2bpL30MjdWrsIlagxeH39sNEdRTkFjS7ZFZGJubon3b9mknDKhHRvVStg8pcxRtBkGo382iKMAiM8qICu/hC4Brgbp718zsqjrCEBfyBLldZcoL8jLR5IkLK1tKpUoP3LkCL/88guvvvoqOTk52vZfeOEFbR9ubm5kZ2ejVqsxNzcnLS2Nxo11l2mjVmrY+UMc1vUs6T6qucllbKmys7n8wiSKz52j0Rtv4PL06KpP0jMZBRlM3jUZ64YOjJ73AXs/+YxfFr5Hz2cmEtrnEeMaV5wL68fApT3QbQZ0n2lQMcB98WXxis4BhpGGkUcWRiYyMpK9e/dy6dIlAAoLC4mPjyc/P5+8vDweffRRPvzwQ06ePAlULVEeGxtbqaOAMony3bt3o1AoUKlUrFu3jm7dutGxY0d2795NTk4OSqWSjRs3as+pTKIcuEei3MPDw+AS5SplKRbWNpiZmWklygcPHqy9H6GhoRw4cIDk5GSSk5OZMmUKb7311h2OAsDc3JyuXbuyadMmAJ1LlB/+9RKK9AJ6Pt1Cr/INtaEkIYHkYcMpSUjA67NPTcJR5JXmMWnnJErVpXzR+wu8Gwcy7J0F+LVtx65lX7B39fcIjcY4xuWkwrK+kLwPHv8SeswyuGrs/oRs/BrY4eWsWymbypCdhZG5XaI8JCSEyMhILl68SG5uLgMGDCAkJISePXveIVE+f/78Wge4b5coDw0NJSIiggEDBuDt7a2VKO/Tp889EuV79+7VtlEuUR4ZGYmZmdkdEuXfffcdERERpKSk6F2ifPz48URGRqJWKbGyta1Uorwq+vbtS1ZWFlCmXLtw4UICAgIoKChg7NixOrE7Iz6HkztSCeraGN/WpiUSWHjoEMkjRqIpLcFn1Srq9+xpbJNQqpW8uvtVUvJT+LDHhzR1KguuW9nY8tj02YT2HcCxXzfy60cLUJaWGNa4jJNlGU95GTB6I4Tes02P3lGqNRy6pKCzgaaggLJphX/CT7t27cTdxMXF3XNMpnLy8/OFEEKUlpaK/v37iy1btmjLBg4cKBITE++oJ4QQc+fOFa+++qphDb2N4sICkZlwURQXFopFixaJ5cuXG6TfmjxbJTeVYsWs/WLl7AOipEipR6tqzo1Nm0RccGuR+OijojQtzdjmCCGE0Gg04o2YN0Tw8mCxOWFzpXWO/bZJLBn2qPjxjVdFYc4Nwxh3YZsQc92F+CBYiKvnDdNnBRxJUgifGb+J389k1LktyvYXqvIdK48sZLS8+eabtG3bljZt2tC8efMKJcoBtmzZQmhoKMHBwRw8eJCZM2cay2RKi4rK4hU2NvdIlJsK+zfEU3C9mN5jg0xKTVaxbBmZr8+kXng7fH78EUtP/S/sqg7Lzi5jc+JmXgh5gUFNB1VYR5Ik2g14nEGvzuRaSjJr3pzG9Yw0/Rp28kdYNxIaNi9LjXWrOAXbEOyLz0aSoJO/4UapspCgzAONIv0yAK6ehl3YVt1nq7RIxbLXYmjVtTEPjWhuAMuqh6akhIsdI7CLiMDrk4/rJAaoS1QaFQ/99BBhbmF82vPTaiUBZMb/zaZFcxBqNY9Nm41XULDuDRMCPmgJTt7w9Caw0k12Ym0Z/OUBlGoNm6d0qXNb1RUSlEcWMg8sGo0GVUkJVjamt1ahnMsXrqPRCALCGxnblDu4efQYorgY55EjTMZRAJzJPkN+aT4Dmw6sdraYR2BzRs5dSj1HJzbMm835fXt0b9jVs5CfCWFRRncU+cVKTl7OoUugYWNfsrOQeWBRFhcjhMDKyOsA7kfKWQVWtha4+5uWKmpB9F4ka2vq3Uq3NhVi0mIwl8zp1LhTjc5zauTOiPeW4NGsBds+XcKhn9fpNhsv/lbaekDv+9czAIcvXUetEQZLmS1HdhYyDyylxUUggaWNjbFNqRAhBClnFXgHuWBmblp/aoV7o6kX0REzE7t3MekxhLqF1kpy3MbenqdmvUfLrj3Yv341f379yR0ihHUifgd4hEJ9448Q9yVkY2NpRpi3YRb/lWNaT7CMTA1QFhVhaW2DWQ32CDAk2WkF3MwtxSfYgOmN1aA0OZnSlBTsH3rI2KbcwdXCq1y4foGunl1r3YaFpSX9J79KxFMjOLt7BxsXvEPJzcK6GVZ0Ay4fhsA+dWtHR+xPyKa9rws2loZ97mVnYSBkiXLdotFoUJYU3xGvGDJkiHZx490cPHiQkJAQQkNDCQkJYcuWLRXW06VEecpZBQDerUzLWRRExwCYnLPYn7EfgK5etXcWcEuEcOgo+k2aSlrcGda+OZ28a1m1bzDxLxAak3AWV/OKic8qoKuB4xUgOwuDIUuU6xZlyb3xiokTJ7J48eIK64eEhHD8+HFiY2P5/fffGT9+PJoKVv/qUqI89ayCht71qedgOgFkgILoaKz8/LBqYlrS6DFpMbjVcyPQKVAn7bXq1osnZ75LwXUFa2a/xtVLCbVrKH4H2LqAZ5hO7KoL5ZLkho5XgOwsTAJZorzsOmoiUd6pUyTd+j3CgkX/dw7du3fnjz/+qFCivF69eljc2o+7qKgIuFfnSpcS5cWFSq5cyjW5KSjNzZvcPHLE5EYVSrWSg5kH6erZVaeaWT6tQxnx3mLMLS1Z984MEo8frlkDGk2ZswjoVaMtUfXFvvhsXAwkSX43prNCSM/ErL9I9uUCnbbZoIk9XYc2q1MbskR57STKKb6JslTJ4KfHMGToUIKCgjA3N69UohzKFH7Hjx9PSkoKa9as0Tq/cnQpUX457jpCYHLOovDwYURpKfbdTMtZnMw6SaGysM5TUBXh6uXNyLlL2bRwDpsXz6PHuAm07fto1ScCZJ6Em9kmMQUlyiXJm7oaRJL8bvQ6spAkqZ8kSX9LkpQgSdLrFZR/KElS7K2fi5Ik5dw6HipJ0kFJks5JknRakiTTmf/QMbJEec0lytesWUP3Pn3pNWAA58+fv6OfyiTKoUy08dy5cxw+fJh58+ZRWlp6R3lFqZa1/ZabclaBjZ0lbr6mljIbjVSvHrbhVa7BMigx6TFYmFkQ4RGhl/btnJwZ9vb7+Ldrz1/ff8Weld+i0VS8SdYdxO8AJGjaSy921YQErSS5cbTF9DaykCTJHPgceBhIA45KkrRFCKH9yxZCTL2t/otA21sfbwJjhBDxkiQ1Bo5LkrRdCJFTW3vqOgLQF0KWKK+VRPlv69fh06w5zz0/UWsbUKlEeflUGUCrVq2wsrIiLi7ujuO6kigXGkHKOQXerVyM8g2wMoQQFO6Nxq5TJ8xMaCEelMUr2jVqh52l/ha8WdrYMOi1WexduYzjWzeTm5XFIy++hqX1fdKH4/8Er3CwM/4IcV/5FqpGCG6DfkcWHYAEIcQlIUQpsA64n97zCGAtgBDiohAi/tb/M4AsQP+bzBoBWaK8ZuTl5WFvV4/69vYobuSwffv2O8orkyhPSkrSxjKSkpJISEjAx8fnjnN1JVGelZJPcYHS5KagShMTUWZkmFy8IqMgg8TcxDqlzFYXMzNzeoydQI+o8SQcO8T6ObMozLlRceXCbEg/YRJTUFAW3PZ1rWcwSfK70aez8AQu3/Y57daxe5AkyQfwA/6qoKwDYAUkVlA2QZKkY5IkHbt27ZpOjDY0skR5zQgLCyOwaQA9BjzK8xMn0rlzZ23Z/STK9+7dS5s2bQgNDWXw4MF8/fXXODuXLWrStUR5yjkFSOAdZFrO4v8ps/p/KdeEfen7gLqnzNaEsEce47HX3iA7NYU1s6ehSLt8b6WEXYCAwIcNZldllEmSXzdKFpSW6kjT1uYHGAJ8d9vnp4FPK6k7o6IywAP4G4ioqj9ZorzuPAgS5Rq1WlxJjBe517LuKTMVifL17x8VGxYeNYgdNSE5aqxIHDjI2Gbcw5SdU0TfDX2FRqMxeN+Z8X+LL8aPEp+OGypSz566s/C/zwixKEAItdrgdt3N0VuS5NtO112S/G4wAYnyNOD2RG4voOLIIwzn1hRUOZIkOQBbgdlCiEN6sVDmDh4EiXJlSUmlelCmIFF+M6+UrJQ8k5uCUhcUcPP4cZPLgipRl3D4ymGdp8xWF/eAZoycuxQ7Jxc2zHuLuJjdZQUaNSTsLBtVmBl/hcG+hDJJ8simxhtZ6DN19igQKEmSH5BOmUO4Z0spSZKaA87AwduOWQGbgJVCiP/q0UaZ2/jwww8rLbtdjnvkyJF3ZEEZktLisjUSFSnNPvPMM4Y25x4uxylAgE+wie2Gd/AgKJUmF684fuU4Raoig05B3Y2jWyNGvLeYXz+Yz++fLSX36hUi2vshFeeYxBQUlMUr2ng64ljP0mg26M1lCiFUwBRgO3AeWC+EOCdJ0hxJkm7f0WQEsO7WcKicocBDwNjbUmtDkfnXU1pUhIWVNWbmxl8gVREpZxXYOljRwMu+6soGpDA6GrP69bENNa0/o5j0GKzNrWnv3t6odtjY2fPkzHdp1a0XB/77I9u/+wo1FuDfw6h2ARSUqDiZmmPceAV6XpQnhNgGbLvr2Ft3fX6ngvNWA6v1aZvMg4cQGpTFRdg6OBrblArRqDWkxl3HL6QBkomlzBbsjcYuMhLJ0njfTCsiJj2GcPdwbC2MLzNvbmFJ3xdewbGROwfW/0i+SwQDNRYYW5f38CUFKo0w2vqKcow/GScjU0208QoT3ezoalIeJTdVJjcFVfL336iyskxuCiolL4WUvBSDpMxWF0mS6PRwd/o3/pu0HDPWvfUfcrOuGtWmfQnZWFuYEeZjWEnyu5GdhcwDQ+ktTSdLW2N/16uYlLMKJDOJJi2N+0d9NwV7owGw61r3LTh1iTZl1oScBQAJOwlyzGLwpOcouFEmQnglMd5o5uxPyKaDn+Elye9GdhYGQpYorzulxUVYWFlhbl7x7On9JMr/+OMPwsLCaN26Ne3atWPPnj0V1lMoFPTq1YvAwED69u1Lbm5ute1LOafAo6kj1kYMQlZEQXQ0NkFBWLq5GduUO4hJi8HXwRdvB29jm3In8X9C/cY06TKIEXOWYGFlzU/vvk7CUcMnZWblFXPxaoHRp6BAdhYGQ5YorxtCCJTFxffdQvV+EuVubm5s3bqVM2fO8P333/P0009XWG/evHn079+f+Ph4unbtyqJFi6plX2FOCdmXC0wvZTY3l6KTJ7EzsZTZIlURR68cpYunaY12UCvh0p6yLChJwtWrCSPnLqFBEx82L53Hmb8ql9zRB/uMKEl+N7KzMAFkifKy67ifRPkz48YiNBqsbGyZMGEC4eHhtGrVijlz5mjtv59EeVhYGB4eHkCZSGFBQUGFGxtt3ryZqKgooGYS5SnnyjY6MjVnUXjgAGg0JhevOHrlKKWaUqOmzFbI5cNQkneHxIedkzND35pPk5bB7F29DOVtWmT6Zl9CmSR5kIfxBSn/NRLlu5d/Q1ZKxVMUtcXNx58eYyfUqQ1Zorx6EuWhISFcuHiRrj5+LFiwABcXF1QqldYJVUeivJz169fTsWNHLCvIDFIoFFq5EE9PTzIzM6t1X1PPKrB3tsalsf6E8GpDwd5ozB0dsW3Txtim3EF0WjS2FraENzIt9Vvi/wQzS/DvdsdhS2sbIoeN5qe3ZxAX8xchDz+id1OEEOw3oiT53VRrZCFJ0s+SJA2QJEkeiegYWaK8ehLlzQKaknHlKuYWFqxdu5awsDDCwsJqJFEOcObMGWbPns2XX35ZrftVnVXFarWG1PPX8Q52Ncoq5MoQGg0FMTHYde2KZELrUoQQ7EvfR0ePjliZm5b6LfE7wCcSrOvfU+TZPAg3v6ac+P1XvYhk3k3itQKu5hlPkvxuqjuy+BIYB3wiSdJ/geVCCMNOnNeRuo4A9IWQJcqrlCgXQoAQYGamlSg/cuQITk5OjB49utoS5ampqTz55JOsXr0aPz+/Cq/B1dWVa9eu0bBhQ9LT03F3d6/0esu5kpCLsliNj4nttV18Lg61QmFyEh9JuUmkF6TzTLDxV9zfQc5lyIqDPvMqLJYkibD+g/jjiw9JOROLb5u2FdbTFTHxphOvgGqOLIQQO4UQo4AwIBnYIUnSAUmSxkmSZFqpHw8YskR51ahKShACzK2sycvLo379+jg4OJCZmVltifIbN24wYMAAlixZQkRE5RvsDBo0iBUrVgDVlyhPOavAzFzCq4WJpcxG7wVJwq6LaQWRY9LL1G9NL2V2R9m/95Ekbx75EPUcnTj5+xa9m7M/IRsf13o0cTGOJPndVHtaSZIkV2As8BxwEviYMuexQy+W/UuQJcqrplwPytLKirCwMIKCgggODmb8+PHVlij/+OOPSUpK4u2339amLTb8LBAAACAASURBVCsUZUHpcePGadOKZ82axdatWwkMDCQ6Oprp06dXaV/KOQWNA52wsjGtEGBBdDS2bdpg4WxaTiwmLYYApwA87D2MbcqdxO8AJx9oEFhpFQtLS0Ie7s+lE0e5kVm7LXerg0lIkt9NdaRpgY1AHDAT8LirrFrytvr+kSXK646pSpRfz0gX11KTq6xnDIny3Oyb4rPnd4mTO1IM0m91USoUIq5FS5H12WfGNuUO8kvyRejKULH06FJjm3InymIh5roL8VvVz3LBjevigxGPiV3ff6U3c44l60+S/G6q+w6v7sjiMyFEkBDifSHEHekhQggTS2eQqS2mKFEuhKC0uKhaEh/GkChPPXcdMMGU2X37QAjsH+pWdWUDcjjzMCqNyvRSZlP2g/JmtXbFs3NypnlkV87u2UnJzUK9mLMvXoEkQaempvNcVXfc3FKSpBPi1h7YkiQ5AyOEEF/ozzQZQ2OKEuWq0hKERoNlNZyFMSTKU84qcGhgg1Mj05hXLqdgbzTmrq7YtAoytil3EJMeg52lHaFupqV+S/wOMLcG3+o5sbD+gzgfs5uzu3fSbkDNt96tin0J12jt6YhTPdPJFqvuyGJ8uaMAEELcAMbrxyTdIgyQ4iajP8r1oKxMSA+q/JlSKdWkXbiOTysTS5lVqynYtw/7rl2RTGDjnnKEEMSkxRDZOBJLMxPLi4n/E/y6glX1nL5700AaN2vJye2/otHcuwi0LpiKJPndVPdJMpNu+2uQJMmcsn2xTRobGxsUCoXsMB5gSouLMLe0xNzCNF4uQggUCgU2NjZkxOegKtXgbWJTUEWnTqPJzTW5lNmLNy6SVZRlellQ1y+BIqFaU1C3E/bIIHKvXiHp5DGdmnMkqUySvKuJOYvqTkNtB9ZLkvQVIICJwB96s0pHeHl5kZaWxrVr14xtikxtEIL869lYWFmTXVhkbGu02NjY4OXlxeFNyZhbmuHZ3LSyjQpiosHcHLvISGObcgflKbOdPTtXUdPAxO8s+zegd41OC2jfif+19+7RbZ/nnefnAcA7xTspiaRIStRdsnW3rpQt24mlJJaTdJo403SbdqbpTJqZpNNJp93ONml65pw57dmZ2Ut2p5lMZ7a7bbLZtrnaiSXbMgFSkiVZFxO8CdSFJEBRJMEreAfw7h8/UCZpyQQpAr/X5vs5B0cE+PsBX5EAn9/7PM/7fbILi7j66k+p3ndw2eTU+YJaWJLPJ95g8W+A3wH+OSDAGeB7iRK1XKSkpDxy85VBf3ru3ubVP/1zTn3199n21PJ9GJeL9sYgZZvzSUnVZ3c0wGitm4w9u3Hm6jUkyuP3sK1gGyWZernf4jsDhRuhsHpRpzldLnZ//JPUff//oq/jLkUVVcsip66tVwtL8vnEuykvqpT6P5VS/0gp9StKqb9USi1vos5gmIe/qQGA8m07bVbyfgZ7xhi8P6ZdF9R0Tw8TTU3adUENTQ5xo/eGfi6zU2Nw17PoFNQMTz73Aq6UVK7+8mfLImfGkly3egXE7w21SUT+TkSaROT2zC3R4gwrm86mBnJLVpNT9P5NdnbToavLrMcaKKRbveLCvQtEVITj5Xrp4m4dhCcsS/IlkLEqh201z9DsPsf4yPBjy6m/ZVl86OIHNZt4C9z/DcsfKgycAP4aeL+ZkcGwTKhoFH9zI+Xbn7BbykNp9wbJW51JbrFeI15Dbjeu1atJ27zZbilz8Pg95Kbl8kSRZr9P3xlIyYTKpddR9pw6TXh6inffeG3hgxegzhckPzNFC0vy+cQbLDKUUm8AopRqV0p9C3g2cbIMK50+fwcToRHWaRgspqciBFoHtTMOVNPTjNbXk338uFatvFEVpS5Qx5HSIzgdGuXhlQLfa7DhGXAt3ZqmuKKKip1Pcv3MK0QfMkslfjkxS/KNRVpYks8n3mAxEbMn94nIV0XkM4BmVSrDR4nORn3rFYHWASLhqHYpqLFr14iGQmQd16s1tTnYTP9Ev34ts30+GOxYdBfUw9hz6iVCwT58ly4s+Tlu9YboHp7QMgUF8QeLrwOZwL8E9gFfBH4jUaIMBn9zAznFJeSWrLZbyvto9wZxpToo3ZRnt5Q5jLrdkJJC1uHDdkuZgzvgRhCOlOrVyosvZvm/xHrFbDbs3U/u6jVcfQw32jqfvvUKiCNYxDbgfU4pFVJK+ZVSvxnriEr+9HLDikAphb/Jq+WqQilFuzdI+dYCnCn67I4Gy+Ijc98+nLPmiOhAnb+OnUU7KczQayWG7wwUb4O8isd+KofDyZ4XXqSrtYn7t9uW9Bx1bUGtLMnns+C7PdYiu090SoIaPtIE/R2MjwxTvl2/YDHQPcZIcEK7FNR0VxeTPp92s7b7J/pp6GvQLwU1OQLt55dlVTHDzhPPk5KewdVXf7Loc8ORKBdvB7VsmZ0h3kuja8BPROTXReSzM7eFThKRkyLSKiJtIvKHD/n+fxSR67HbTREZnPW93xARX+xmUl4rCH+TF4B12/WaGw1WCgr0a5kNua3d0bq1zNYH6lEo/Vxmb9dCdHrJ+yseRlpmFjuefo6W8x5GBwcWde4N/yChybC2KSiIP1gUAEGsDqgXY7dPfdAJsfTVd4BTwHbgCyIyxwJTKfV7SqndSqndwP+GNTcDESkAvgkcBJ4CvhlzujWsADqbvWQXFmlZr+hoDFJQmsWqAn2MDQFCHg8pZWWkbthgt5Q51AXqKEgvYHuhXu63tJ2F1FVQ8eipiUthz8kXiUbC3Dj76qLOe2BJvkGvi5DZxGX3oZT6zSU891NAm1LqNoCI/AB4CWuI0sP4AlaAAHgBOKuU6o+dexY4CXx/CToMHyKsekUDlU/s1qr9E2BqIkyXb5Bdz62zW8ocolNTjF64QN6nX9LqZxaJRqjvqufp8qdxiEb1HaUsS/LqE+BcXoPKgtIy1u/Zz42zv+CpT38OV0p8z1/f1sfO0lzys/T1Z413B/d/E5G/mn9b4LQyoHPWfX/ssYc9fyWwHnhzMeeKyJdF5IqIXDFmgR8N+rv8jA0NarkZz98yQDSitEtBjV+5ghobI0uzekVDXwNDk0P61St6mmA4sKwpqNnsPXWasaFBbl7wxHX86GSYqx0DHNukbwoK4k9D/Rx4JXZ7A8gBQguc87BLnEd5hb8M/N0sv6m4zlVKfVcptV8ptf9hc5cNHz7eq1foV9xu9wZJTXeyplovg75QrRtJTSXroF5mi56AB4c4OFyqVyvvg5bZZdhf8TAqn9xDQdk6rv7ip3GNR7h0p59wVGldr4D4jQT/ftbtb4DPAQt9mv3A7PV6OdD1iGNfZm6KaTHnGj5CdDY1kJ1fQN6aUrulzGGmZXbd9gKcTo1SKlgWH5kHD+LI0Mt6xOP3sLt4N7lpegVXfGdhzZOQszYhTy8i7Dn5Ivdvt9HV2rzg8R5fH2kuB/s0sySfz1Lf9ZuAhZqTLwObRGS9iKRiBYT37VgRkS1APjB76+NrwMdFJD9W2P547DHDRxilFP5mL+Xbn9Aq9w4QDIwyOjhJhWYWH1MdHUzduaNdy2zvWC/N/c36dUGND0LHxYSloGbYcfxZ0rKy4tqkV9/Wx4Eq/SzJ5xNvzWJERIZnbsDPsGZcPBKlVBj4KtYf+Wbgh0qpRhH5toicnnXoF4AfqFnrtVhh+8+wAs5l4NszxW7DR5fB7i5GB/q13IzX7rV21+rmB/WgZVYzi4+6gOV+q50l+e1zoCLLur/iYaSkp/PEsy/gu3Se4b5H11N7RiZovT+ifb0C4u+GWrWUJ1dKvQq8Ou+xP5l3/1uPOPevgIWK6IaPEJ2x+RXrduhX3G73Bilal01W3tIN5xJByF1LamUlqZWVdkuZgyfgoTijmC35W+yWMhffWUjPg7L9CX+pPS98ind+/mOun3mF4//4Sw895nybtW9H93oFxL+y+IyI5M66nycin06cLMNKxN/kJTM3j/y1D22as42J0Wm6bw9r1wUVHR9n7O1LZGm2EW86Os2FrgvUlNfolU6MRq1gsfE5cMY7JHTp5BSXsPHAIRpe/yXTkxMPPcbj6yNPU0vy+cRbs/imUmpo5o5SapD39kQYDI+NUorOpgYt6xWdzf2oqKJyp15Xf2OXLqEmJ7Wbine95zqh6ZB+LbPdN2C0J+H1itnsPXWaidEQzXVvve97M5bkR6v1tCSfT7zB4mHHJT40G1YMQ/e7CfUHtZxf0dEYJC3Lxer1el39hdweJCODzAOJT6kshrpAHS5xcWjt8u6Ofmx8rwMC1c8l7SXLtu2guGoDV199fxvtrd5RuocntPaDmk28weKKiPwHEakWkQ0i8h+BdxIpzLCy6GyO1Ss021+hoor2xn4qthdqdfWnlCLkdpN16BCONL3qKJ6Ah72r95Kdqpf7Lb4zULYXspO3J0tE2HvyRYL+Djq8N+Z8r77Napqo+RAUtyH+YPEvgCng/wV+CIwDv5soUYaVh7+xgYycXArK9LLS6O0cYXx4Srt6xdSdu0x3dmpnHNg92o1vwKdfCmo0CP7LSU1BzbD16NNk5OS+r422rq2PigJ9LcnnE2831CjwPtdYg2G56Gz2sm7bTu3qFe3eIAhUbC+wW8ocQu5aALJr9Pqj7AlYrbza7a+49SagEt4y+zBcqansev4kF3/0Qwa775G3Zq1lSX4ryKd26bX59IOItxvqrIjkzbqfLyJmk5xhWRjquc9IX6+W8yvavUFWV+WQsUovg7dRt5u0TRtJKdOrc8zj91CaVcqGXL3cb/GdgcwiWLvHlpff9bFP4HA4uPbLnwFwwz/EiOaW5POJNw1VFOuAAkApNYCZwW1YJmb2V+hmHjgemuL+XQ1bZkdHGb18RTvjwKnIFBfvXdSwZTYCba9bqwqHPVYt2QWFbD50DO9bZ5kcG6O+rQ8ROFKt13vrg4j3JxcVkQf2HiJSxaNNAQ2GReFv8pK+Koei8scfb7mcdDT2g0I7i4/Rixdhelq7ltl37r/DeHhcv3pF4CqM99uSgprN3k+cZmp8nMbaN6j7EFiSzyfe9tc/BupEpDZ2/zjw5cRIMqw0/M0NlG/dgdh01fco2r1BMlalUFKxJAODhBGqdePIyiJzrz0plUfhCXhIcaRwYM0Bu6XMxXcGxAEbTtgqY+3GLazdtIWrv/gZ17Jf4p/UVNuqZ7HE6zr7S2A/0IrVEfX7WB1RBsNjMdzXw1DPfe1aZqNRRUdTkIodhYiOLbNHjiBxDtZJFh6/hwNrDpCZoll3j+8MlD8FmfY3Kew5dZqh+12Uhdo/VPUKiL/A/U+x5lj8fuz2fwPfSpwsw0phZn6FbvWKnrvDTI6GtatXTN70Ee7u1q5ltnO4k7vDd/VLQY3ch3vXbU9BzbD54FGiGTnsHmlgf5XeluTziXfd/zXgANCulDoB7AHMaDrDY9PZ5CU9K5viiiq7pcyh3WvNRF63zf6r0dnMtMxm1egVLLRtmW173frXhv0VD8PpcnG7aBfrxjoZvR+wW86iiDdYTCilJgBEJE0p1QJoZidp+DDib2qgbJue9Yo11bmkZ+mV6hl1e0jbto2U1Xo1I9YF6qhYVUFljl7ut7Sdhew1sEaPlWvPyATnpBqcrrhmXehEvJ9Qf2yfxY+BsyLyE8zkOsNjMtLfx+D9e9r5QY0OTdLbMaJdCioyMsLY1avaDTqaCE9wqfuSfquKSBja3rRSUJq08p5vCzLhzKBs31Ga3OcYD43YLSlu4i1wf0YpNRibPfE/Af8VMBblhsfiQb1Cs2FHHY3WnC3dgsVo/XmIRLSrV1zuvsxkZFK/eoX/EkwOaZOCAsviIy8zhWc++yuEpybxvnnGbklxs+i1v1KqVin1U6XUVCIEGVYOnU0NpGVmUVy13m4pc2j3BsnKTaWwTC8jvJDbjSM3l4wnn7Rbyhw8AQ/pznT2r9HL/RbfGXC4YMMzdisB5lqSr1m/gXXbn+Daaz8nGonYLS0u9EoUG1YU/iYvZVu343DoM3s4EonS2dxP5c5CrXYhq2iUkMdN9tGjiEuf6QBKKTx+DwfXHiTNqZf7Lb6zUHEY0vWwlr/dN8q9ofcsyfecepGRvl7arly0WVl8mGBhsIXQQD8D9wLatczevz3E1HhYu0FHE83NRHr7tEtB3R2+iz/k1y8FNRSA+169UlA+y5J8Zn9F9f6D5BSv5uqrH45CtwkWBlvwz8zb1ixYtHuDOJxC+Va9euBH3W4QIevYMbulzMHjt1pmj5XrpYu2s9a/OgWLtj7WFWRQUWhtWnQ4nOx54ZMEWhq5f+eWzeoWxgQLgy34m72kZmRQUqWXO2m7N8jajbmkZuiT6gHL4iN9505chXoV3T0BDxtyN1CWrZf7Lb6zkLsOivXo8J+xJJ+/a3vnsx8nJS2dax+CNloTLAy20NnYQNmW7Tic+tQrRvonCAZGqdyhVwoqPDDA+I0b2rXMjk2PceX+Ff1SUOFJuP2WVi2z7wZmLMnnTulLz8pm+9PP0VJfy+jggE3q4sMEC0PSGR0coL/Lr129oqMxCGjYMltXD0ppV6+4eO8i4WhYv/0VHRdgKqRVCqreZ1mSH36IJfneUy8SCYd59/Vf2qAsfkywMCQdf3MjoGe9YlVBOvlr9TLCC7ndOAsKSN+p134UT8BDpiuTvSV77ZYyF99ZcKbCen2Cq6etjx2lORQ8xJK8oLScqt37uHH2VSLhaRvUxUdCg4WInBSRVhFpE5GHjmUVkc+JSJOINIrI3856/M9jjzWLyP8qOvUxGh6LzqYGUtLSKVmvj0VzZDqKv2VAv5bZSITRujqya45pZYmilKIuUMfh0sOkOPWyRMF3FqqOQWqW3UoAGJ0Mc61j4EHL7MPYe/JFRgcHuHmhLonKFkfC3n0i4gS+A5wCtgNfEJHt847ZBPwRcFQptQP4euzxI8BR4ElgJ5aJoV6TXgxLxt/UQNnW7Tg12i/QdWuQ6cmIdimoCa+XyMCAdlPx2gbb6B7t1q9eMXAX+lq1SkFdutvPdERRM69eMZuqXXvJX1vG1V/8FKX0nCuXyEuVp4A2pdTt2G7vHwAvzTvmt4HvxMa0opTqiT2ugHQgFUgDUoD7CdRqSBJjw0ME/R3aWXy0e4M4XQ7KtujVMhuqdYPDQfbRo3ZLmcOMy+yxMs1aZn36tczW+/pIdTk+0JJcHA72nHqR7ls+7vlakqgufhIZLMqAzln3/bHHZrMZ2Cwi9SJyUUROAiilLgDngHux22tKqeYEajUkCX+znvMrOrxByjbnkZKmT3cWWPWKjN27cebl2S1lDh6/hy35W1idtdpuKXPxnYWCDVCoT4qzrq2PA1X5pKd88Htrx9PPkZaZpe0mvUQGi4clfuevr1zAJuAZ4AvA90QkT0Q2AtuAcqwA86yIvG8dLiJfFpErInKlt9eM1/gw4G/y4kpLY031RrulPGCod5yB7jEqNEtBhfv6mPB6tWuZHZka4VrPNf26oKbH4Y5bq1VFz8gELd0jH1ivmCE1PYOdJz7GzbfrGQn2JUHd4khksPAD62bdL+f9tuZ+4CdKqWml1B2ssa2bgM8AF5VSIaVUCPgFcGj+CyilvquU2q+U2l9c/Oh8oEEf/E0NlG7ehtOlT1FU15bZkMcqdurWMnuh6wIRFdGvXnG3HsLj2kzFA7hwy3pvxTtCdc/JT4GC62deSaSsJZHIYHEZ2CQi60UkFXgZmL+++jFwAkBEirDSUreBDuBpEXGJSApWcdukoT7kjI8M09txl3Ua1itySzLIK9GtZbYWV3ExaVu32i1lDp6Ah1Wpq3iyWC/3W3xnwJUBlfrUUep8liX5jtLcuI7PLVlD9f6nePeN15iemkywusWRsGChlAoDXwVew/pD/0OlVKOIfFtETscOew0IikgTVo3iG0qpIPB3wC2gAbgB3FBK/SxRWg3Jwd9i7a8o36FPvSI8FcHfOkDlDr1WFSocZrSunqzjNVq18kZVlLpAHUdKj+By6NPNhlLge83aW5GSbrca4D1L8iPVhTgd8f8O95w8zcTIMC11tQlUt3gS+ttWSr0KvDrvsT+Z9bUC/lXsNvuYCPA7idRmSD7+Ji+ulFTWVG+2W8oDAjcHiUxHtUtBjV+/TnRkhOzjenWMt/S30Dfep18KKnjLaps9/FW7lTzgdt8oXUMT/G6cKagZ1u14guKKKq7+4qfsPPExbS4W9NnlY/jI09nUwNrNW3Gl6FOvaPcGcaU4KN2sV7dRqNYNLhdZRw7bLWUOMy6zR8v0auXFF5s4p1G9or5triV5vIgIe06dpq/jLp2NDYmQtiRMsDAkhYlQiN72O1pZfCilaPf2Ub41H9cCbY3JJuTxkLl3L85Vq+yWMoe6QB07CndQlKGX2SJtZ6FoC+RX2a3kAXU+y5K8snDxO8m3Hnua9FU5XNXIjdYEC0NSCLQ2glKUb9enuD3UM85w34R2Kajp+/eZbGnRrgtqcGKQd/ve1a9ldmoU7tZptaoIR6JcuP1+S/J4SUlNY9fzJ7n1ztsM3u9eZnVLwwQLQ1LobGzAmZLC2o16zBcAKwUFUKFZcTvkdgNot7/ifNd5oiqqX73ijhsiU1rtr3g3MMTIRDiu/RWPYtfHP4GIcP01PXp7TLAwJAV/s5e1m7bgSn2/66ZdtHv7yF+bRU5Rht1S5jDqduMqXUvqRn02LoLVMpufls+Owh12S5mL7wykZlvztjWhPjZC9Uj10oPFqoIiNh88SsObZ5maGF8uaUvGBAtDwpkcG6Xnzm3Kt+lTr5iaCBPwDWqXglJTU4zWnyf7+HFtumAAItEI9YF6jpYdxenQqL6jlGXxseEZcOlzIVL3AZbki2HvJ04zNT5GY+0by6Rs6ZhgYUg4gZYmlIpqVdwOtA4QDSvtgsXY1atEx8a0a5ltDDYyMDmgXwqqtwWGOrVKQY1NhbnaMcCxTY/fBLB201bWVG/i2i9+hopGl0Hd0jHBwpBwOpsacLpcrN2sV70iJd3J2ur4dtYmi1CtG0lJIevQQbulzMET8OAQB0dKj9gtZS4atsxeumNZki+1uD0bEWHvqdMM3Atw98bVZVC3dEywMCQcf1MDazZuJiU1zW4pwEzLbJB12wpwuvT6CITcbjKfegpHpl7WIx6/hyeLniQvXa/9KPjOwuonIKfUbiUPqItZkh+oKliW59t8+BhZ+QW2t9Hq9UkxfOSYGh/j/p1bWqWg+rtGCQ1MamfxMeX3M3XrFtnH9Ur19I330Rhs1G92xcSQNW970/N2K5lDXVsf+ysXtiSPF6crhV0fO8XdG1cJBjoXPiFBmGBhSCiB1mZUNKpVcVv3llndpuLVB+oB9NtfcfstiIa1qlf0jkzS0j2yLPWK2ex6/hROl4trv7CvjdYEC0NC6WxqwOF0UrpZH+fUdm+QwvJssvP1SIvNMFrrJqWigtSqKrulzMET8FCUUcTWAn1+h4BVr0jLhfKn7FbygPO3lmbxsRCZuXlsPfo0je43mBgNLetzx4sJFoaE4m9qYE31ZlLS9XACnRwP031rSLsuqOjkJKNvv61dy2w4GuZ813mOlR3DIRr9uVAKfK/DxmfBqY/7bZ2vj9yM+C3JF8OeU6cJT07iffPMsj93PGj02zd81JiaGOf+7TatLD78zf1Eoxq2zF66jJqY0M7i493edxmZGtGvZba7AULdWqWglmpJHi+r11dTtnUH1157hWg0suzPvxAmWBgSRldrM9FIRKthR+3eIGmZLtasz7FbyhxCbjeSnk7mgQN2S5mDJ+DBKU4Ol+qzOxp4r2V2oz7F7TsxS/LHsfhYiL2fOM1w731uXXk7Ya/xKEywMCQMf7MXcTgo3brdbinArJbZ7QU4nHq99UPuWrIOHsShSbpuBo/fw56SPaxK1cv9Ft9ZKN0D2SV2K3nAjCV5zTIXt2ezcf8hVhUV29JGq9cnxvCRorPJy5oNm0hN18N7qa8zxNjwlHYpqKm7d5lu7yBLsxTU/dH7tA606tcFNdYP/ktapaDAapktz8+goiBxe2QcTid7XvgU/iYvPXdvJ+x1HvraSX01w4phenKC7rabWtUrHrTMbtcrWOjqMlsXqAPQr15x601QUa2CRTgS5fwty5I80Q0KTzz7Aq60tKSvLkywMCSErpstRCNhrTbjtXuDlFSuIjNHH8M5sCw+UqurSS0vt1vKHDwBD2uy1rAxTy/3W3xnIbPQSkNpQsMyWJLHS3p2NttrTtBSX8vY8FDCX28GEywMCcHf7EXEQekWPeoVE6Fp7t/RsGV2bIyxS5e0W1VMR6a50HWBmrIarVp5iUatqXgbnweN3G9n6hXJCBYAe0+dJjI9zbuv/zIprwcmWBgSRGdjAyXrq0nTxOOoozmIUlChWbAYvfg2anpaO4uPqz1XGQuP6Wfx0XUNxoKwUR/jQFg+S/J4KSyvoPLJPdw48wqRcDgpr2mChWHZmZ6apLutlXU79EpBpWenUFKpW8tsLY7MTDL27bNbyhw8fg8uh4tDaw/ZLWUuvjOAwMbn7FbygLGpMO+0Dyz7ru2F2HvqNKGBfm6+XZ+U1zPBwrDs3LvZSiQcplyT/RUqquho7KdiRwGOBGyWWipKKctl9shhHBpNEASrXrF/9X4yU/RYGT7AdwbKD0Dm8ji6LgczluTJSkHNsH73PvLXlnLt1eQUuk2wMCw7/uYGEKFMk/0VPe0jTISmtatXTN26Rbjrnnb1ikAowO2h2/p1QYV6rTSURl1QYNUrUp3LZ0keL+JwsPuFF7nX1so9X2vCX88EC8Oy42/yUlK1gfSsbLulANasbRGo2KZXsAjVatoy64+1zOq2v+LWG4DSatARQF1bkP1V+WSkJr/gvvOZ50jNyEhKG21Cg4WInBSRVhFpE5E/fMQxnxORJhFpFJG/nfV4hYicEZHm2PerEqnVsDyEp6bo8rWwTrP9FavX55KenWK3lDmE3G7StmwhZc0au6XMwRPwUJ5dTlVOFVZEpgAAFmdJREFUld1S5uI7A9mrYc2Tdit5QF9okuZ7w0lPQc2QmpHJnpMvkpGTg1Iqoa+VMLtGEXEC3wE+BviByyLyU6VU06xjNgF/BBxVSg2IyOy9+38N/Dul1FkRyQbsHUBriIvutptEpqcp367HB3pseIqe9hEOnt5gt5Q5REIhxt55h8Lf/E27pcxhMjLJ2/fe5jObPqNXy2wkDG1vwNZPgUOfhMhMy2yyi9uzOfby/5CU10nkT/0poE0pdVspNQX8AHhp3jG/DXxHKTUAoJTqARCR7YBLKXU29nhIKTWWQK2GZaIzVq8o37rDbikAdDRZu7Z1q1eMnj8P4bB2LrNXuq8wEZnQr14RuAITg9qloOrb+shJd7GzTK9Z7okgkcGiDJg9A9Afe2w2m4HNIlIvIhdF5OSsxwdF5B9E5JqI/EVspWLQHH9TA8UVVaRn61KvCJKZm0rROj30zBByu3GsWkXG7t12S5mDJ+AhzZnGgTV6ud/iOwPihOoTdit5gFKKOl8fR6qLEmJJrhuJDBYP++nNT6q5gE3AM8AXgO+JSF7s8RrgXwMHgA3Al973AiJfFpErInKlt7d3+ZQblkQkPE3XzVZtLD6ikSidTf1U7ijUKqWilGK01k3WsaOIS5/BPWDtr3hqzVOku/Ryv8V3BioOQ7o+V/B3g2N0DU0s+whVXUlksPAD62bdLwe6HnLMT5RS00qpO0ArVvDwA9diKaww8GNg7/wXUEp9Vym1Xym1v7i4OCH/CUP8dLf5CE9NamMe2H1nmMmxsHYpqMmWFsK9vWQff9puKXNoH26nY6RDvy6o4S5r2JFmKag6n3WBame9IpkkMlhcBjaJyHoRSQVeBub3d/0YOAEgIkVY6afbsXPzRWQmAjwLNGHQms6mBgDKNKlXtHuDOBxC+TZ9NnDBrJbZGr2sNDx+D4B+Fh9tr1v/6hYs2vooy8ugslCzjYsJImHBIrYi+CrwGtAM/FAp1Sgi3xaR07HDXgOCItIEnAO+oZQKKqUiWCmoN0SkASul9V8SpdWwPPibvRRVVJGZo0eqoN0bZE11LmkZeqV6Qm436Tt24CrS64rUE/BQlVPFulXrFj44mfjOQE4ZlOixyRMgElVJsyTXhYR+ipRSrwKvznvsT2Z9rYB/FbvNP/csoEf/pWFBIuEwgdYmdj6jx9VfaGCSoD/E4c9U2y1lDpGhIcavX6fon/2O3VLmMDY9xpXuK3x+6+ftljKXyDTcegue+BXQ6I/yjCX5SqlXgNnBbVgm7t/2EZ6c1GYznrYts/X1EI1qt2v7cvdlpqJT+rXMdlyEqREtLT4AjlTr9f5KJCZYGJaFziYvgDbmge3eINn5aRSUZtktZQ6hWjfOvDzSn9CjY2wGT8BDhiuDfav1cr/FdwYcKbBer2YAj6+X7WtzKMxOs1tK0jDBwrAs+Ju9FJZXkJmbZ7cUIuEonc39VO7UrGU2GiXk8ZBVU4M49dk2pJTC4/dwaO0hUp16ud/iOwtVRyFNn30yY1NhrrYPrqgUFJhgYVgGopEIgZYmbVYV924NMT0R0S4FNdHYSKS/X7sU1O2h23SNdunXMjvYAb3N2qWgLt8dYCoStc0Pyi5MsDA8NvfvtDE9Ma7NsKN2bxCHSyjbkm+3lDmEat0gQtaxo3ZLmcNMy6x29QrfWetfzYLFjCX5U0m2JLcbEywMj01no7W/QpeVRbs3SNmmPFLT9WuZzdi1C1e+XkHME/CwKX8Ta7L0cr/Fdxbyq6Bwo91K5uDx9bGv0h5LcjsxwcLw2PibveSXlpOVZ/8fweG+cQbujVK5U68UQTgYZKKhQTvjwNBUiKv3r+q3qpiegDu11qpCo7rTjCX5SqtXgAkWhsdkpl6hTctso6Yts3V1oBRZmtUrLt67SFiF9QsW7fUwPaZdCur8Lev9tdLqFWCCheEx6bl7m6nxMco1MQ9s9wbJKUontyTDbilzCNW6cRYVkb5tm91S5uAJeMhOyWZXyS67pczFdxZc6VCll/VIvc+yJH9iBViSz8cEC8Nj4Y/5Qa3ToF4Rno7gbxmgcqdeFgwqHCZUX092TQ2i0eAepRR1/joOlx4mxaHXFEF8Z6CqBlL0CfpKKeraVo4l+Xz0eecaPpR0NnvJX1tKdoH9aZ8u3yDh6ah2KajxdxuIDg1pV6+4OXCTnvEe/VJQwVvQf0u7FNTd4BiBwXGOrsB6BZhgYXgMotEIgeZGrbqgnCkOyjbbvzFwNiF3LTidZB05YreUOXgCxmV2MdRpMELVTvTqLTR8qOhtv8vk2Kit9QqlFMFAiHZvEN/l+5RvycelQUujUopJn49Rj4ehv/8HMvfswZmTY7cslFLcGb6Dx+/hh60/ZFvBNoozNZkFE7xlpZ/e/kso3AQF6+1WBEBHcIy3bvbw38/fpSwvg6oVYkk+HxMsDEvGb5Mf1NR4GH/LAO3ePtq9QUaHpgAorljF/k9WJVXLbKKjo4y+/TahWjcht5vwvXsApG3ZQtG//Be26RoPj3O5+zJuv5u6QB2BUACAjXkb+drer9mmi+kJaK+zitm+M9B/23q8aDM8/y3bZE2GI1y+M8C51h7OtfZwu3cUgMrCTP7w1Fat6mHJxAQLw5LpbGogd/UacooSe2WqlGLg3hjt3iDtjX3caxsiGlGkpjtZt72Ayp2FVOwoJCs3uaZuSimm7t5l1O0mVOtm7PJl1PQ0jsxMso4eIeuf/zOyjx8nZU3yN7t1DnfiDrjxBDxcvmc5yma4Mji45iC/tfO3OFZ2jNLs0qTrYqDdCgxtr8PtWgiPW11P64/Doa/AxudtWVF0DY7zVmsv51p7qG/rY2wqQqrLwcH1BXzxYCUntpawvkgvU8pkY4KFYUmoaJRASyMbDxxKyPNPT0YItA5YAcIbZKR/AoCC0ix2PbeOyp2FrKnOxelMbtktOjHB2OXLD1YP0x0dAKRWV5P/xS+S/fRxMvfuRVKTa8g3FZniyv0rePwe6gJ13B2+C0BVThWf2/I5aspr2Ld6H2nOJLukhqeg44IVIHxnoa/Vejy/Cvb+ulXErjqW9K6n6UiUq+0DnGvt5a3WHlq6RwAoy8vgs3vLOLGlhMPVhWSmmj+RM5ifhGFJ9HW2MxEaWdYU1GCPtXro8AYJ3BwkEo7iSnOybms++05VUrGjkFUF6cv2evEy5fcTqq0l5HYz9vYl1MQEkp5O1qFDFHzpN8g+fpzU8vKk67oXuocn4MET8PD2vbcZD4+T6kjlwNoDvLz1ZWrKaqjIqUi6Loa73kst3X4LpkLgTIXKo7DvS1bhunBj0ndm9wxP8NZNKzh4fH2MTIRxOYQDVQX8j5/YyoktJWwsyV6xaaaFMMHCsCRm5m2ve4zidng6QtfNwVh6KchQzzgAeasz2fl0GZU7CyndmIczJcmrh6kpxt9558HqYeq2lUtPqagg71d/lezjx8l86gCOtORepU9Hp7necx2P3woQbYNtAJRll/FS9UvUlNdwYM0BMlxJ3psQCYP/UixAnIX71nuDnHJ44let1cP640m3GY9EFdc7B3krVnvwBoYBWJ2TxiefWMszW0o4urGQVema7THRFBMsDEvC3+Qlp3g1OcUlizpvuG+cjkYrteRvHSA8FcWZ4qB8Sz67nl1HxY5CcouTvxFr+t49Qm4PIY+bsfMXiI6NIampZB44QP7Ln7dWD1VVSdfVM9ZDfaAeT8DDha4LhKZDuBwu9q3ex6c3fpqa8hrW56xP/tVwqMeqO/jOwK03YWIIHC6oOAwf+7YVIIq3Jn310D86hfumVXuovdnL4Ng0DoF9lfl844UtnNhSwra1q8zqYQmYYGFYNCoaxd/sZcPeAwseGwlHuXdr6EHtYeCe1VmSU5TOtiOlVO4spGxzXtLbXdX0NOPXrxOKFacnb94EwFW6lpyXTpN9/DhZBw/iyExum2Q4Gqahr+HB6qGlvwWAkswSXqh6gZryGg6tPURWSpKLrdEIdF2L1R7OWF8DZK+GbS9awWHDM5CeXBuMaFTh7RriXEsvb93s4XrnIEpBUXYqz21dzYmtxdRsLCY306weHhcTLAyLJujvYHxk+JH1itDApLV6aAzS2dzP9EQEh0so3ZjH9qNrqdxZSN7qzKRf3YV7ewl56gi53YzW1xMdGQGXi8x9+yj5xjfIfvo4qdXVSdfVP9FvrR78Huq76hmeGsYpTnaX7Obre79OTXkNm/I2Jf9qeKzfWjXMdC+NBUEcUP4UPPtvrQCx5smkrx6GxqbxtPVyrqWX2ps99IWmEIFd5Xl8/bnNnNhazM7SXBwr0JIjkZhgYVg0nc3W/oqZYUfRSJTuO8MPVg9BfwiA7Pw0Nh9Yba0etuQnfb6EikQYf/ddRj0eQrVuJhobAXCVlJBz8gWyamrIOnIEZ3Zyc+lRFaUp2PRg9eDt86JQFKYXcmLdCWrKazhcepic1CRv4otGofvd94rTgSugopBZCBs/ZhWmq5+FzOQO/VFK0dI9wrnWHt5q6eWdjgEiUUVeZgrHNxVzYmsxxzcVr6h52HZggoVh0fgbG8guKKLrVpSLP/XS2dTP5FgYcQhrq3M5/JlqKncWUlCalfzVw8AAo3V1hGrdjNbVERkcBIeDjD17KP693yP76eOkbdmSdF1Dk0Oc7zr/YPXQP9GPIDxZ/CRf2f0Vaspr2FawDYck2YFnYghunbMCRNtZCN0HBEr3wPE/sFYPpXsgyQaIockwdb4+3mrt4a3WXrqHrdbpnWU5fOWZap7ZUsLudXkr0tDPLkQpZbeGZWH//v3qypUriz6v6aKb1/7Tf06Aoo8uURXCkbqF1KxTuKaHyB5pJHukiaxQM87ohG26HFEoGrAMz0YyoaVKaFkPrVXCeLp9f1SiAvdcEBVhVUSxbxz2jcOeCciN2iYLB4qyaBdOogyTzTuuPVxy7eeKaw+DDvv8taJK0dE/xnREsSrNRc3mIp7ZUsIzm4spyUl+6/RHHRF5Rym1f6HjVvzKIiU9E6eY5eticEo62dOjZPf8OSnTXQgK5YJQHtj9lvJtdXJrvYv7qx0Pcum5QK6d10QK9o2lsGsqgw3hNBxYuiLp0G+jLICmlON4sw5xN30bUbGaDIpjNzt5fvtqTmwpYV9lPilJ3nhpeDgJXVmIyEngfwGcwPeUUv/+Icd8DvgWoIAbSql/POt7OUAz8COl1Fc/6LWWurIwGAyGlYztKwsRcQLfAT4G+IHLIvJTpVTTrGM2AX8EHFVKDYjI/Kb9PwNqE6XRYDAYDPGRyPXdU0CbUuq2UmoK+AHw0rxjfhv4jlJqAEAp1TPzDRHZB6wGziRQo8FgMBjiIJHBogzonHXfH3tsNpuBzSJSLyIXY2krRMQB/M/ANxKoz2AwGAxxkshq5MPaT+YXSFzAJuAZoBzwiMhO4IvAq0qpzg9qcRSRLwNfBqiosMEwzWAwGFYIiQwWfmDdrPvlQNdDjrmolJoG7ohIK1bwOAzUiMhXgGwgVURCSqk/nH2yUuq7wHfBKnAn5r9hMBgMhkSmoS4Dm0RkvYikAi8DP513zI+BEwAiUoSVlrqtlPo1pVSFUqoK+NfAX88PFAaDwWBIHgkLFkqpMPBV4DWs9tcfKqUaReTbInI6dthrQFBEmoBzwDeUUsFEaTIYDAbD0ljxO7gNBoNhJRPvPouPTLAQkV6g/TGeogjoWyY5y4nRtTiMrsVhdC2Oj6KuSqXUgpv2PzLB4nERkSvxRNdkY3QtDqNrcRhdi2Ml6zKmKwaDwWBYEBMsDAaDwbAgJli8x3ftFvAIjK7FYXQtDqNrcaxYXaZmYTAYDIYFMSsLg8FgMCyICRYxROTPRORdEbkuImdEpNRuTQAi8hci0hLT9iMRsW+E2SxE5FdFpFFEoiJie3eIiJwUkVYRaRMRbXb7i8hfiUiPiHjt1jKDiKwTkXMi0hz7HX7Nbk0ziEi6iFwSkRsxbX9qt6YZRMQpItdE5Od2a5mNiNwVkYbY366EbTYzweI9/kIp9aRSajfwc+BP7BYU4yywUyn1JHATa/6HDniBzwJuu4XMmp1yCtgOfEFEttur6gH/HThpt4h5hIHfV0ptAw4Bv6vRz2sSeFYptQvYDZwUkUM2a5rha1huFDpyQim1O5HtsyZYxFBKDc+6m8X7HXJtQSl1JmadAnARy5DRdpRSzUqpVrt1xIhndootKKXc2D89dQ5KqXtKqauxr0ew/gDOHx9gC8oiFLubErvZ/lkUkXLgk8D37NZiFyZYzEJE/p2IdAK/hj4ri9n8FvALu0VoSDyzUwwPQUSqgD3A2/YqeY9Yuuc60AOcVUrpoO0/AX8ARO0W8hAUcEZE3omNbUgIKypYiMjrIuJ9yO0lAKXUHyul1gF/g2WCqIWu2DF/jJU++BuddGlCPLNTDPMQkWzg74Gvz1tZ24pSKhJLB5cDT8Vm3NiGiHwK6FFKvWOnjg/gqFJqL1Ya9ndF5HgiXiSR8yy0Qyn1fJyH/i3wCvDNBMp5wEK6ROQ3gE8Bz6kk9jov4udlN/HMTjHMQkRSsALF3yil/sFuPQ9DKTUoIm9h1XzsbBA4CpwWkU8A6UCOiPw/Sqkv2qjpAUqprti/PSLyI6y07LLXElfUyuKDEJFNs+6eBlrs0jKb2KjZfwOcVkqN2a1HU+KZnWKIIdb4yf8KNCul/oPdemYjIsUzHX8ikgE8j82fRaXUHymlymPzdV4G3tQlUIhIloismvka+DgJCqwmWLzHv4+lWN7F+oHr0k74vwOrgLOx1rj/bLcgABH5jIj4saYaviIir9ml5VGzU+zSMxsR+T5wAdgiIn4R+Sd2a8K6Uv514NnYe+p67KpZB9YC52Kfw8tYNQutWlU1YzVQJyI3gEvAK0qpXybihcwOboPBYDAsiFlZGAwGg2FBTLAwGAwGw4KYYGEwGAyGBTHBwmAwGAwLYoKFwWAwGBbEBAuDYRGISGjhoz7w/L8TkQ2xr7NF5C9F5FbMYdUtIgdFJDX29YraNGvQGxMsDIYkISI7AKdS6nbsoe9hmQxuUkrtAL4EFMXMEN8APm+LUIPhIZhgYTAsAbH4i9hGzgYR+XzscYeI/B+xlcLPReRVEflHsdN+DfhJ7Lhq4CDwb5VSUYCYa+4rsWN/HDveYNACs8w1GJbGZ7HmLewCioDLIuLG2h1dBTwBlGDtKP+r2DlHge/Hvt4BXFdKRR7x/F7gQEKUGwxLwKwsDIalcQz4fswh9T5Qi/XH/Rjw/ymlokqpbuDcrHPWAr3xPHksiEzN+P4YDHZjgoXBsDQeZov+QY8DjGO5lgI0ArtE5IM+g2nAxBK0GQzLjgkWBsPScAOfjw3qKQaOYxm51QG/EqtdrAaemXVOM7ARQCl1C7gC/GnMBRYR2TQzK0RECoFepdR0sv5DBsMHYYKFwbA0fgS8C9wA3gT+IJZ2+nus+Rpe4C+xJtANxc55hbnB458Ca4A2EWkA/gvvzeE4Abya2P+CwRA/xnXWYFhmRCRbKRWKrQ4uYU0y647NZzgXu/+owvbMc/wD8EcazTk3rHBMN5TBsPz8PDbAJxX4s9iKA6XUuIh8E2s+eMejTo4NcPqxCRQGnTArC4PBYDAsiKlZGAwGg2FBTLAwGAwGw4KYYGEwGAyGBTHBwmAwGAwLYoKFwWAwGBbEBAuDwWAwLMj/D+aWe9MhAcnDAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "accuracy_s1 =np.array(accuracy_s).reshape(len(C_s),len(gamma_s))\n",
    "x_axis = np.log10(C_s)\n",
    "for j, gamma in enumerate(gamma_s):\n",
    "    pyplot.plot(x_axis, np.array(accuracy_s1[:,j]), label = ' Test - log(gamma)' + str(np.log10(gamma)))\n",
    "\n",
    "pyplot.legend()\n",
    "pyplot.xlabel( 'log(C)' )                                                                                                      \n",
    "pyplot.ylabel( 'accuracy' )\n",
    "pyplot.savefig('RBF_SVM_Diabetes.png' )\n",
    "\n",
    "pyplot.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从图中大致可以看出log(C) = 4时，log10(gamma) = -4,即C = 10000,gamma=1e-4, accuracy值最大，"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 使用GridSearchCV对RBF核SVM进行正则超参数调优\n",
    "在sklearn框架下，不同学习器的参数调整步骤相同： 设置候选参数集合 调用GridSearchCV 调用fit"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GridSearchCV(cv=5, error_score='raise',\n",
       "       estimator=SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,\n",
       "  decision_function_shape='ovr', degree=3, gamma='auto', kernel='rbf',\n",
       "  max_iter=-1, probability=True, random_state=None, shrinking=True,\n",
       "  tol=0.001, verbose=False),\n",
       "       fit_params=None, iid=True, n_jobs=1,\n",
       "       param_grid={'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]},\n",
       "       pre_dispatch='2*n_jobs', refit=True, return_train_score='warn',\n",
       "       scoring='neg_log_loss', verbose=0)"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.svm import SVC\n",
    "\n",
    "#需要调优的参数\n",
    "# 请尝试将L1正则和L2正则分开，并配合合适的优化求解算法（slover）\n",
    "#tuned_parameters = {'penalty':['l1','l2'],\n",
    "#                   'C': [0.001, 0.01, 0.1, 1, 10, 100, 1000]\n",
    "#                   }\n",
    "penaltys = ['l1','l2']\n",
    "Cs = [0.001, 0.01, 0.1, 1, 10, 100, 1000]\n",
    "# tuned_parameters = dict(penalty = penaltys, C = Cs)  #GridSearchCV要求dictionary参数形式\n",
    "tuned_parameters = dict(C = Cs)  #GridSearchCV要求dictionary参数形式\n",
    "\n",
    "SVC_penalty= SVC(kernel='rbf', probability=True)\n",
    "grid= GridSearchCV(SVC_penalty, tuned_parameters,cv=5, scoring='neg_log_loss')\n",
    "grid.fit(X_train,y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Python36\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('mean_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Python36\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split0_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Python36\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split1_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Python36\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split2_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Python36\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split3_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Python36\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('split4_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n",
      "C:\\Python36\\lib\\site-packages\\sklearn\\utils\\deprecation.py:122: FutureWarning: You are accessing a training score ('std_train_score'), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True\n",
      "  warnings.warn(*warn_args, **warn_kwargs)\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'mean_fit_time': array([0.03081794, 0.0302175 , 0.02962461, 0.02921696, 0.04102917,\n",
       "        0.09566989, 0.16371627]),\n",
       " 'mean_score_time': array([0.00200434, 0.00179844, 0.00200372, 0.00180178, 0.00180116,\n",
       "        0.00159898, 0.00180206]),\n",
       " 'mean_test_score': array([-0.49324991, -0.49358949, -0.49175968, -0.48721034, -0.52483146,\n",
       "        -0.58096024, -0.59870391]),\n",
       " 'mean_train_score': array([-0.44405915, -0.44308874, -0.44243318, -0.38986629, -0.35347677,\n",
       "        -0.40861092, -0.45692746]),\n",
       " 'param_C': masked_array(data=[0.001, 0.01, 0.1, 1, 10, 100, 1000],\n",
       "              mask=[False, False, False, False, False, False, False],\n",
       "        fill_value='?',\n",
       "             dtype=object),\n",
       " 'params': [{'C': 0.001},\n",
       "  {'C': 0.01},\n",
       "  {'C': 0.1},\n",
       "  {'C': 1},\n",
       "  {'C': 10},\n",
       "  {'C': 100},\n",
       "  {'C': 1000}],\n",
       " 'rank_test_score': array([3, 4, 2, 1, 5, 6, 7]),\n",
       " 'split0_test_score': array([-0.5134772 , -0.51512294, -0.51426315, -0.49259046, -0.51850061,\n",
       "        -0.56557451, -0.5930368 ]),\n",
       " 'split0_train_score': array([-0.43657207, -0.43852101, -0.43805667, -0.39017446, -0.34512274,\n",
       "        -0.41461188, -0.45197381]),\n",
       " 'split1_test_score': array([-0.48802461, -0.49330132, -0.4903162 , -0.49391526, -0.52770417,\n",
       "        -0.60924538, -0.64525809]),\n",
       " 'split1_train_score': array([-0.4497107 , -0.44812082, -0.44721435, -0.38837203, -0.34283474,\n",
       "        -0.42418941, -0.4787879 ]),\n",
       " 'split2_test_score': array([-0.53948066, -0.53468545, -0.5273648 , -0.52077405, -0.53882823,\n",
       "        -0.60616761, -0.61364405]),\n",
       " 'split2_train_score': array([-0.4346635 , -0.43298975, -0.43176894, -0.37784493, -0.36461096,\n",
       "        -0.39065618, -0.43865498]),\n",
       " 'split3_test_score': array([-0.43794836, -0.44126035, -0.43850967, -0.44627285, -0.48040496,\n",
       "        -0.54940507, -0.58634026]),\n",
       " 'split3_train_score': array([-0.45485118, -0.45427378, -0.45342426, -0.40388603, -0.39168304,\n",
       "        -0.42036941, -0.46768716]),\n",
       " 'split4_test_score': array([-0.48665101, -0.48288991, -0.48769565, -0.48208079, -0.55868485,\n",
       "        -0.57422238, -0.55482918]),\n",
       " 'split4_train_score': array([-0.4444983 , -0.44153834, -0.44170169, -0.38905399, -0.32313236,\n",
       "        -0.39322771, -0.44753348]),\n",
       " 'std_fit_time': array([0.00214277, 0.00039538, 0.00049273, 0.00116743, 0.00127237,\n",
       "        0.0073417 , 0.01406393]),\n",
       " 'std_score_time': array([5.52884409e-06, 3.98813310e-04, 5.59222454e-06, 4.00209498e-04,\n",
       "        3.99874418e-04, 7.97149267e-04, 3.99994918e-04]),\n",
       " 'std_test_score': array([0.03365369, 0.03160433, 0.03038661, 0.02405708, 0.02586274,\n",
       "        0.02328595, 0.02997045]),\n",
       " 'std_train_score': array([0.00765422, 0.00742542, 0.00744417, 0.00829523, 0.02318411,\n",
       "        0.01397125, 0.01442442])}"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# view the complete results (list of named tuples)\n",
    "grid.cv_results_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.4872103372976179\n",
      "{'C': 1}\n"
     ]
    }
   ],
   "source": [
    "# examine the best model\n",
    "print(-grid.best_score_)\n",
    "print(grid.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "如果最佳值在候选参数的边缘，最好再尝试更大的候选参数或更小的候选参数，直到找到拐点。c=1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "the correct classification fraction of the zero_one_loss of SVC on test is 0.7402597402597403\n",
      "the correct classification fraction of the zero_one_loss of SVC on train is 0.8306188925081434\n"
     ]
    }
   ],
   "source": [
    "from sklearn.metrics import zero_one_loss  #评价SVC预测模型的性能\n",
    "# 预测\n",
    "y_test_pred_rbf_svm = grid.predict(X_val)\n",
    "y_train_pred_rbf_svm = grid.predict(X_train)\n",
    "# 使用zero_one_loss评价模型在测试集和训练集上的性能，并输出评估结果\n",
    "#测试集zero_one_loss(y_true, y_pred, normalize=False)\n",
    "print ('the correct classification fraction of the zero_one_loss of SVC on test is', 1 - zero_one_loss(y_val, y_test_pred_rbf_svm))\n",
    "#训练集\n",
    "print ('the correct classification fraction of the zero_one_loss of SVC on train is', 1 - zero_one_loss(y_train, y_train_pred_rbf_svm))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "使用GridSearchCV发现的最佳参数做预测，测试集上正确分类的比例是0.7402597402597403，训练集上正确分类的比例是0.8306188925081434"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
